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PREFACE 
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tutes for Research (AIR) were obtained through contract No. F41609-C- 
0036. 

Dr. Robert W. Stephenson was principal investigator of the techni- 
cal work that resulted in the conduct of the literature review. Mr. 
Clifford P. Hahn was coinvestigator. Dr. James R. Burkett, Technical 
Training Division, was the Air Force contract monitor. 

The authors gratefully acknowledge the many contributions of 
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SECTION I 
INTRODUCTION 



This is one of several reports describing the work conducted 
during a two-year study. One of these reports (Stephenson and Burkett, 
1973)^ describes a systems analysis of the Air Force OJT program. One 
additional report (in two volumes), a •'Program Evaluation Guide** for 
evaluating on-the-job training programs, is planned. 

The Air Force on-the-job training (OJT) system is vast and com- 
plex, both from a training and a management standpoint. The present 
OJT system encompasses a wide diversity of technical specialties and 
command missions, and the system must provide training to a broad mix 
of trainees having varying aptitudes and levels of Air Force exper- 
ience. A myriad of training materials, directives, and instructional 
approaches are used. Because of its diversity and complexity, the 
OJT process is expensive and difficult to manage. It is also under- 
stan«»ably difficult to evaluate in terms of training adequacy and 
cost effectiveness. 

Tfhere is a need for better and more easily applied methods of 
monitoring and evaluating the Air Force OJT program in terms of costs, 
quality and effectiveness of training provided, determination of op- 
timal OJT-resident school mix, and timeliness in incorporating inno- 
vative OJT instructional techniques as they become available. While 
it is recognized that actual job performance is the ultimate criterion 
of training success, performance test development and administration 
are often too costly to be practical in so vast a training system as 
Air Force OJT. There is, therefore, a need for exploration of new 
techniques that could improve OJT effectiveness. 

Additionally, there is a need to compile the availably 
information about OJT into a single useful source document with rec- 
ommendations for areas where the Air Force OJT system can be most 
easily and rapidly improved. 



^Stephenson, Robert W. (American Institutes for Research) and 
Burkett, James R. (Air Force Human Resources Laboratory). "'Analysis 
of the Air Force On-The-Job Training System, Draft Technical Report 
prepared for Technical Training Division, Air Force Human Resources 
Laboratory, Lowry Air Force Base, Colorado (in press). 
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OBJECTIVES OF THE PROJECT 



Because of these needs, an Air Force research requirement was 
formulated and modifie- as a result of extensive consultations with 
a variety of interested headquarters locations. The objectives of 
the investigation were finally stated as follows: 

(1) Describe the operation, capabilities, and functional 
requirements of OJT as presently conducted throughout 
the Air Force through appropriace application of systems- 
analysis and cost-analysis techniques, 

(2) Identify and document problems being experienced by OJT 
managers, trainers, and trainees in the field in accom- 
plishing representative OJT programs, 

(3) Devise and demonstrate new methods and alternatives for 
measuring the effectiveness of OJT programs, and rec- 
ommend ways to apply these methods to improve OJT through- 
out the Air Force (Statement of Work, Contract No, F41609- 
72-C-0036, p,l). 



PROJECT PHASES 



The decision was made to conduct the work in two phases. The 
first phase would be concerned with problem definition, the second 
phase with the evaluation of several innovative approaches. 

Specific objectives of Phase 1 were: 

(1) Review of civilian and military technical literature on 
OJT as well as all current Air Force regulations, manuals, 
and other documents governing or describing the OJT system, 

(2) Conduct of a systems analysis of the Air Force OJT program 
to describe what the system is supposed to be, what its 
programmed functions are, how it is supposed to opefate 
under current directives, and how it relates to other 
systems and personnel programs, 

(3) Determine, through interviews and questionnaire responses, 
what problems are being experienced in the field in con- 
ducting representative OJT programs. 
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(4) Determine and document the extent of correspondence and 
interaction between OJT and formal resident training pro- 
grams, 

(5) Make recommendations on how to improve the present Air 
Force OJT system based upon the results of che overall 
systems analysis accomplished in objectives (1) through 
(4), above (Statement of Work, Contract No. F41609-72- 
C-0036, p. 1). 

The present report describes the results cf the literature re- 
task (Objective 1). 



SECTION II 



SCOPE OF LITERATURE REVIEW 

The requirements for the Literature Review, to be conducted as 
part of this project, were stated as follows: 

'•During Phase I, the contractor shall conduct and 
document a thorough, comprehensive review of the 
literature pertaining to on-the-job training meth- 
ods, materials, and media. Topics re/iewed shall 
include, but not be limited to, on-the-job train- 
ing, correspondence technical training, apprentice- 
ship training, field training, and operational 
training. The contractor shall also be responsible 
for reviewing and synthesizing all applicable Air 
Force directives, regulations, and manuals on the Air 
Force on-the-job training program as well as any 
parallel or pertinent Army, Navy, or other DOD 
OJT directives and policy documents*' (Statanent of 
Work, Contract No. ril609-72-C-0036, pp. 1-2). 

References were selected in accordance with this requirement, and 
were briefly reviewed by the senior author. Many of the references 
were literature reviews and bibiliographies assembled by other organ- 
izations, so the total number of citations that were considered for 
inclusion numbered in the thousands. In addition, approximately SO 
government documents and manuals were obtained by the contract moni- 
tor and forwarded to the principal investigator for possible inclu- 
sion in the study. 

SOURCES 



Relevant references were gleaned from many sources. The liter- 
ature review covered a period from 19S7 through 1973 and was brief in 
scope, but thousands of references were considered. Computer searches 
were made of relevant abstracts that are contained in the Educational 
Resources Information Center (ERIC) and the Defense Documentation 
Center (DDC) data bases. Special computer lists of unpublish^ work 
currently in progress (funded by the Department of Defense) were also 
obtained and searched. These computer runs produced abstracts of 
approximately 2,000 relevant reports, documents, and current projects. 

Other important sources of references were the existing bibli- 
ographies and literature reviews and abstracting services in the area 
of vocational education. These have been produced on a recurring basis 



for Biiany years by the Ohio State Univer^-ty Center for Vocational 
and Technical Education. ^ The focus of most of these references is 
upon problems of civilian education rather than military education, 
but many useful citations were found to be applicable to this re- 
search. Other bibliographies were reviewed as useful sources of 
information (e.g., bibliographies of reports produced by organiza- 
tions that specialize in training research, such as the Human Re-- 
sotirces Research Organization (HumRRO), the Naval Personnel and 
Training Research Laboratory, the Air Force Human Resources b^borr 
tory, etc.). 

In addition to these computer searches and secondary sources 
of literature reviews, special independent reviews were conducted 
of recOTt jcumals that were likely to have information about train- 
ing innovations and research* The Training and Development Journal 
of the American Society for Training and Development, for example, 
was a ''gold mine** source of information. It contained abstracts 
of articles that were selected because they were considered useful 
for people in the training field. Other relevant sources included 
such publications as the Journal of Applied Psychology j and Personnel 
Psychology. 

The training literature is not very inspiring. Campbell (1971)' 
conducted a review of the personnel training and development litera- 
ture in 1971 and summarized his views as follows: 

**By and large, the training and development literature is vol- 
uminous, nonempirical , nontheoretical, poorly written, and dull . . . 
it is faddish to the extreme. The fads center around the introduc- 
tion of new techniques and follow a characteristic pattern. A new 
technique appears on the horizon and develops a large stable of ad- 
vocates who first describe its 'successful* use in a number of situa- 
tions. A second wave of advocates busy themselves trying out numer- 
ous modifications of the basic technique. A few empirical studies 
may be carried out to demonstrate that the technique 'works.' Then 
the inevitable backlash sets in, and a few vocal opponents begin to 



Center for Vocational and Technical Education. Abstracts of 
Research and Related Materials in Vocational and Technical Education 
(ARM), a quarterly publication of the ERIC Clearinghouse on Voca- 
tional and Technical Education. 

'Campbell, John P., **Personnel Training and Development,'* 
in Mussen, P. and Rosensweig, M. (eds.). Annual Review of Psychology, 
Vol. 22, 1971, pp. 565-566, Annual Reviews, Inc., Palo Alto, Calif., 
1971. 



criticize the usefulness of the technique, most often in the absence 
of -^ata. Such criticism typically has very little effect. What does 
have an effect is the appearance of another new technique and a rep- 
etition of ^e cycle. 

••Anol.her recurring phenomenon is the cyclical article. That 
is t^ere are several prototype papers that appear in the literature 
at regular intervals. For example, there is the article admonishing 
people to evaluate their training efforts. There is another which 
argues that training should be well -planned and systematic. Yet 
another reminds us that training and development efforts must have 
the support of top management or they will fail. There are 5-10 
of these basic types, which say the same things over and over again 
in almost the same language.** (Campbell, 1971)** 

We found the literature on OJT to be essentially as Campbell 
describes it. Most references were quickly rejected on the grounds 
that they contained nothing new or that they were concerned with 
formal classroom training rather than OJT. We were not, however, 
only concerned with experiments that proved something. Our own lit- 
erature review was much more tolerant than Campbell's, since we also 
sought manuals and procedural guidelines that could be used by non* 
experts to evaluate training programs. We were also interested in 
new ideas, whether they had been proved valid or not. 



Reprinted, wi^h permission, from •'Personnel Training and 
Development, Annua]. Review of Psychology ^ Volume 22, pages 565- 
566, Copyright, 1971 by Annual Reviews, Inc. All rights reserved. 
Permission to quote is also given by the author, John P. Campbell. 



CONDUCT OF THE LITERATURE REVIEW 



The work started by gathering bibliographies and literature 
reviews. These were reviewed, and those citations and abstracts 
that were deemed interesting or pertinent were collected. During 
a period of several months, the selected references were obtained. 
The senior author reviewed the mass of references and chose those 
works that were to be included for bibliographic citation. Each 
reference was identified and placed into one or more of the follow- 
ing categories; 

Literature Reviews and Bibliographies 
Handbooks and Manuals 
Training Cost-Effectiveness Literature 
Technique Comparison Studies 
Systems Analysis of Training 
Approaches to Program Evaluation 
Military DocuTients 
Innovations 

The report is organized accordingly. A selected bibliography accom- 
panies each of the following sections of this report. Wherever ab- 
stracts were available, they have been included. An author index 
is also provided. 
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SECTION III 



LITERATURE REVIEWS AND BIBLIOGRAPHIES 



The training literature is frequently reviewed, and bibliog- 
raphies (annotated and otherwise) exist in great numbers. Some 
indication of the mass of the literature can be gleaned from the 
large number of citations (449 in one reference and 213 in another) 
cited in very selective literature reviews of tJie work performed 
during recent years (Smith, 1967;^ Campbell, 1971^), 

The Ohio State Center for Vocational and Technical Education 
is a regular source of references in this area, since they publish 
literature reviews regularly under contract with the U.S. Department 
of Health, Education, and Welfare, Office of Education. In addition 
to these general sources, a number of specialized literature reviews 
and bibliographies concerned with specific subject-matter areas were 
searched. 

The literature and state-of-the-art reviews and bibl iograhies 
that seemed most relevant for OJT are presented in this section. 

BIBLIOGRAPHY 

Barlow Esther. Abetvaate of Personnel Reaeopoh Reporte: VIII. 1954- 
1968, Technical Report AFHRL-TR-68-124, Air Force Human Resources 
Laboratory, Personnel Research Division. liackland AFB, Texas, 
December 1968. A]>-695 A83. 

Begle, Elsie P., Dunn, James A.. Kaplan, Robert M., Kroll, John, 

Melnotte, Judith M., Steel, Lauri. Career Education: An Anno- 
tated Bibliograpny for Teaohere and Currioulm Developers, 
American Institutes for Research, Palo Alto, Calif., January 
1973. 

Bergman, B.A. and Siegel, A.I. Training Evaluation and Student 

Achievement Measurement , A Revieu of the Literature, Technical 
Report AFHRL-TR-72-3, Technical Training Division, Air Force 
Human Resources Laboratory, Lowry AFB, Colorado, January 1972. 
AD-7A7 040. 



5 Smith Robert G., Jr. An Annotated Bibliography on the Design 
of Instructional Systems, Technical Report 67-5, The George Washington 
University, Human Resources Research Office, Alexandria, Va., May 1967. 

^Op. cit., Campbell 1971. 
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Bilodeau^ E. A. Principles of Skill Acquisition, Acad^nic Press, 
New York, 1969. 



Briggs, L. J. Sequencing of Instruction in Relation to Hierarchies 
of Competence J Monograph 3, American Institutes for Research, 
Pittsburgh, Pa., 1968. 

Campeau, P. L. ••Selective Review of Literature on Audiovisual Media 
of Instruction," in Briggs, L. J. et al .. Instructional Media: 
A Rpocedure for the Design of Multimedia Instruc'^ion, a 
Critical Review of Research, and Suggestions for Future 
Research, Monograph 2. American Institutes for Research, 
Pittsburgh, Pa., pp. 99-142, 152-176. 

A review of educational literature considered most pertinent to the 
selection and use of appropriate audiovisual media for achieving 
given learning requirements is presented. Three types of lesearch are 
reported: comparative effectiveness studies (comparison of a new 
medium to conventional methods or to other new media); utilization 
studies (comparing methods of using a given medium); and basic studies 
(highly analytical studies that seek to explore media and learner 
variables that may be related to achievement). Included are studies 
conduct^; in educational institutions from elementary school through 
the university. Excluded are production studies, military research, 
attitudinal and motivational re<?earch, and media-preference research. 
Research literature of the three types is reported for each of the 
following: television; motion pictures; programmed instruction 
(primarily linear, paper and pencil programs); filmstrips, slides, 
transparencies, an?^ other pictorial presentations; radio and record- 
ings; three-dimensional models; and field trips. Limitations of 
current educational research and a suggested strategy for future 
research on media conclude the chapter. — Auth* 

Chalupsky, Albert P., and Kopf, Thomas J. Job Performance Aids and 
Their Impact on Manpauer Utilization, WDL TR 3276 (prepared 
for Office of Manpower Policy, Evaluation and Research, Depart- 
ment of Labor) WDL Division, Philco-Ford Corp., Palo Alto, Calif., 
May 1967. (PB 175 608) 

Childs, Gayle B. "Review of Research in Correspondence Study," in 
Wedemeyer (ed.). The Brandenburg Memorial Essays on Corres- 
pondence Instruction: II. University of Wisconsin Press, 
Madison, Wis., 1966, pp. 126-140. 

Research is not a notable emphasis in the field of correspond- 
ence study. There is, however, a persistent and continuing effort 
on the part of some institutions to carry on research programs, and 
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a notable improvement in research studies. The next five years may 
bring increased recognition of the importance of research to corres- 
pondence education. — Auth. 

Christal, R.E. "Implications of Air Force Occupational Research for 
Curriculum Design," in Snith, B.B. and Moss, J., Jr, (edsO, 
Report of a Seminar: Ppocess and Techniques of Vocational ^ 
Cnrrieulm Development, Minnesota Research Coordinating Unit 
for Vocational Education, University of Minnesota, Minneapolis, 
1970- 

Cunninghain, J,W. ^'Ergometrice'' : A Systematic Approach to Some 
Educational Problems j Center Monograph No, 7, Center for 
Occupational Education, North Carolina State University, 
Raleigh, North Carolina, 1971. 

Cunningham, J.W. (ed.). The Job-Cluster Concept and its Curricula^ 

Implications: A Symposium^ Center Monograph No. 4, Center for 
Occupational Education, North Carolina State University, 
Raleigh, North Carolina, 1969. 

DeCrow, Roger and Grabowski, Stanley M. (eds.). Research and Inves- 
tigation in Adult Education: 1970 Annual Register^ ERIC 
Clearinghouse on Adult Educaton, Adult Education Association 
of the USA, Washington, D.C., 1970. (ED 045 866) 

Embracing 750 items on adult education research and investiga- 
tion (mostly dated 1969 or 1970), this annotated bibliography covers 
adult learning characteristics, program planning and administration, 
learning environments, instructional methods, curriculum materials 
and instructional devices, personnel and staffing; education of 
specific clientele groups, special program areas, professional and 
technical continuing education, management and supervisory develop- 
ment, occupational training, labor education, institutional sponsors, 
and international perspectives in adult education. — Auth. 

Ebel, Robert L., Noll, Victor H., and Bauer, Roger M. Encyclopedia of 
Educational Research ^Fourth Edition), A Project of the American 
Educational Research Association, The MacMillan Co., Collier- 
MacMillan Limited, London, 1969. 

Finch, G. (ed.).. Symposium on Education and Training Media j National 
Academy of Sciences, National Research Council, Washington, 
D.C., 1960. 

Foley, J. P., Jr. Job Performance Aids Research^ Summary and Recomfnenda- 
tions. Air Force Human Resources Laboratory, Wright -Patterson 
AFB, Ohio, April 1969. (AD 697 034) 
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Foley, J. P., Jr. Performcmoe Testing: Testing fov What Is Real, 

Meoorandum AMRL P-42, 6S70th Aerospace Medical Research Ubora- 
tories. Aerospace Medical Division, Wright -Patterson AFB, Ohio, 
June 1963. (AD 412 776) 

Galanter, E.H. (edO» Automatia Teaching: The State of the Art, John 
Wiley, § Son, New York, 1959. 

Geis, G.L. Behavioral Objectives: A Selected Bibliogr^hy and Brief 
Revieujj ERIC Clearinghouse on Media and Technology, Stanford. 
Calif., April 1972. 

The purpose of this paper is to identify and organize the major 
topics in the area of behavioral objectives and to present a sampling 
of the literature in that area. (Note: Where a reference is made to 
the same source in two different sections, the refer^ice is repeated 
in each bibliography for the convenience of the reader.) 

No attempt has been made to produce a comprehensive bibliography. 
This Herculean task fortunately has been carried out by others. The 
Canadian Teachers' Federation (1971) has recently made available a 
large and comprehensive bibliography of books, articles, and papers 
totaling more than 350 items. Poulliotte and Peters (1971) have 
produced an even larger bibliography. It is not annotated, but it 
is divided into several sections for easier searching and is an in- 
valuable source to the student. 

A nioiber of books contain articles or chapters discussing be- 
havioral objectives. The most comprehensive collection is a recent 
book by Kapfer (1971), which includes about 40 separate articles on 
such topics as: "Behavioral objectives and the teaching of values," 
"Behavioral objectives and the educational technologist," and "Clas- 
sifying objectives to improve instruction. "--ilMt/i. 

Glaser, Robert (ed.). Teaching Machines and Programmed Learning. II. 
Data and Directions^ DAVI, National Education Association, 
Washington, D.C., 1965, 832 pp. 

Glaser, Robert (ed.). Training Research and Educaton, University of 
Pittsburgh Press, Pittsburgh, 1962. 

One of the essential obligations of a society at peace is the 
education of its members. It is also true that an essential mission 
of a military establishment during peacetime is the training of its 
personnel. Both education and training have a common basis in the 
findings of psychological research. This book examines training 
research accomplished by experimental psychologists, and considers 
the implications of the research for education in general. 
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A good portion of the work performed by the individuals involved 
in this effort has been devoted to research and development in prob- 
lems of training and the underlying phenomena of learning that are 
involved. The results of this endeavor are obviously relevant not 
only to specialized military problems, but to civilian education and 
to the science of learning. A unique aspect of this work is the fact 
that an increasing immber of persons trained in experimental psychology 
have been turning their attention to the problcas of training and edu- 
cation. In the light of the expressed concern of the United States 
with education, this meeting of science and education is of great 
interest. The purpose of this volume is to present a representative 
account of the training research that has been performed and to examine 
its implications for psychological research and for training and 
education. 

Goodman, Edith H. (ed.). Automated Education Handbookj Automated 
Education Center, Detroit, Michigan, 1965. 

Grabowski, Stanley M. (ed.). Research and Inoestigation in Adult 
Education: 1971 Annual Register^ ERIC Clearinghouse on Adult 
Education, Syracuse, New York; and Adult Education Association 
of the USA, Washington, D.C., 1971. 

This is an annotated bibliography including 578 items of research 
or investigation in adult education, mostly dated 1970 or 1971. They 
cover adult learning characteristics, program planning and administra- 
tion, learning environments, instructional methods, curriculum mater- 
ials and instructional devices, personnel and staffing, education of 
particular groups, program areas, professional and technical continu- 
ing education, management and supervisory development, labor education, 
occupational training, institutional sponsors, and international per- 
spectives in adult education. — Auth* 

Hansing, Ruth A. and Matlock, Eugene W. A Survey of Trainings-Related 
Utilizationa of Television, Research Report SRR 70-19, Naval 
Personnel and Training Research Laboratory, San Diego, Calif., 
February 1970. (AD 701 698) 

This report provides a broad survey of the current state of television 
training technology in civilian and military Situations including 
detailed information concerning a variety of specific applications 
at local Naval training facilities in the San Diego area. The rela- 
tive effectiveness of the existing conventional and unique uses of 
television is studied and recommendations are made regarding the 
feasibility of applying existing systems to other Navy contexts or 
to developing modified or innovative systems. 
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The research information presented includes data pertaining to 
specific equipments, personnel requirements, cost, and training factors 
associated with typical television systems. 

The effectiveness of television, as a potent and diversified 
training tool, is confirmed in a wide variety of research findings 
and demonstrated in a broad range of operational training situations. 
The most prevalent training use of television is simply as a device 
to present filmed instructions in a classroom. The more effective 
utilizations of television, however, appear to be those applications 
that capitalize on the unique capabilities of the medium, such as in 
providing immediate performance feedback or in supplying visually 
programmed task direction. — Auth. 

Hie key, A. E. (ed.). Computer-Aaaiated Instruction: A Survey of the 
Litevature (3rd ed.)* Entelek, Inc., Nei^buryport , Mass., 1968. 

Householder, D. L. and Suess, A. R. Reoiew and Synthesis of Research 
in Industrial Arts Education (2nd ed.). ERIC Clearinghouse, 
The Center for Research and Leadership Development in Vocational 
and Technical Education, The Ohio State University, Columbus, 
Ohio, October 1969. (ED 034 898) 

Impellitteri, Joseph T. and Finch, Curtis R. Review and Synthesis of 
Research on Individualizing Instruction in Vocational and 
Technical Education^ Information Series No. 43 (VT 013 338), 
ERIC Clearinghouse on Vocational and Technical Education, 
Columbus, Ohio, December 1971. 

In the reporting of individualized instruction research and de- 
velopnent activities, priority was placed upon investigations that 
dealt with instructional problems within the general parameters of 
vocational and technical education. Other studies were discussed 
outside those parameters when it was perceived to be necessary for a 
more adequate coverage of the topic under discussion. Individualized 
instruction efforts in areas highly related to vocational and techni- 
cal education (military training, industrial training, and other 
areas of education) were thus included. No attempt was made, however, 
to prepare a thorough review of research in those related areas. 

The materials used in this review and synthesis were gathered 
from several places. A number of useful reports were identified 
through the computer search conducted by the ERIC Clearinghouse at 
Ohio State University. As a result of contacting state Research 
Coordination Unit directors, several other meaningful items were 
obtained. Dissertation Abstracts served as a prime source for rele- 
vant doctoral theses. Contacts with key personnel in military train- 
ing research resulted in the inclusion of many meaningful research 
efforts from this sector. — Auth. 
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Keller, Claudia Merkel. Criterian^Refevenoed Measurement ^ a Bibli- 
ography, TM Reports No. 7, ERIC Clearinghouse on Tests, 
Measurement, and Evaluation, Educational Testing Service, 
Princeton, N.J., January 1972. 

Kerlinger, Fred N. (ed.)- Revieu) of Research in Education, F. E. 
Peacock, Itasca, Ill», 1973. 

Critical and synthesizing essays on four areas of educational 
research: learning and instruction; school organisation, effective- 
ness, and change; research methodology—multivariate analysis; and 
history of education. — H.G* 

Kurtz, Albert K. and Smith, Mary C. Annotated Bibliography of Human 
Factors LcC'^oratory Reports (1946-1968) . Technical Report 
NAVTRADEVCEN IH-158, Naval Training Device Center, Human Factors 
Laboratory, Orlando, Florida, February 1969, 369 pp. 

Lapp, Diane. 272^ Use of Behavioral Objectives in Education, ERIC/ 
CRIER and the International Reading Association, The Interna- 
tional Reading Association, Newark, Delaware, 1972. 

A behavioral objective has been defined by R. P. Mager as a 
statement telling the conditions under which a specified behavior 
will occur (external conditions), the type of behavior that is to 
occur as a result of plarmed instruction (terminal behavior), and 
the performance level that will be accepted (acceptable performance). 
The term behavioral objective has accumulated the following syno- 
nyms that appear frequently in educational literature: goals, in- 
structional objectives, educational objectives, performance objec- 
tives, and training objectives. 

This paper, which accepts Mager 's definition, begins by review- 
ing the literature dealing with behavioral objectives. The litera- 
ture is divided into four categories: 1) educational significance 
of behavioral objectives, 2) evaluation of student learning and of 
the instructional program through behavioral objectives, 3) student 
awareness of and participation in behavioral objectives, and 4) teach- 
er training in the development and use of behavioral objectives. 

The section that follows is a synthesis of the strengths and 
weaknesses of the research reviewed. The final section proposes 
recommendations for future research, including a thinking process 
proposed by the author for use by the classroom teacher when planning 
and evaluating the instructional program. — Auth. 



17 



Larson, Milton E. Revieu and Synthesis of Research: Analysis for 
Curriaulm Development in Vocational Eduoationy Information 
Series, ERIC Clearinghouse on Vocational and Technical Educa- 
tion, The Ohio State University, Columbus, Ohio, October 1969. 
(ED 035 746) 

Lewis, Wiley B. Review and Analysis of Curricula for Occupations in 
Construction^ Information Series No. 24 (VT Oil 753), ERIC 
Clearinghouse on Vocational and Technical Education, The Center 
for Vocational and Technical Education, The Ohio State University, 
Columbus, Ohio, October 1970. 

References believed to be of value to persons desiring informa- 
tion relating to the planning of curricula for new occupations in the 
construction industries were identified through a search of both ERIC 
publications and non-ERIC publications. ERIC publications included: 

Abstracts of Instructional Materials in Vocational and Technical 
Education TAIM), Fall 1967-Spring 1970. 

Abstracts of Research and Related Materials in Vocational and 
Technical Education (ARM), Fall 1967-Spring 1970. 

Current Index to Journals in Education^ VoL I, Vol. II, Numbers 1-3. 

Manpower Research: Inventory for Fiscal Years 1966 and 1967. 

Manpower Research: Inventory for Fiscal Year 1968. 

Research in Education (RIE) Volumes I-IV; Volume 7, Numbers 1^7. 

The three non-ERIC sources of information that were searched were: 

CIRF Abstracts. The International Vocational Training Information 
and Research Centre, C/O The International Labor Organization, 
Volume I-VIII; Volume IX. First Dispatch, Geneva, Switzerland, 
(Fe^>ruary 1970). 

Corplan Associates of I IT Research Institute. A Bibliography of 
Published and Unpublished Vocational and Technical Education 
Literature, Research Coordinating Unit, Vocational and 
Technical Education Division and State of Illinois, Board 
of Vocational Education and Rehabilitation, Illinois, 1966. 

Ehrenreich, Julia W. (ed.). Education Index, The H.W. Wilson 
Company, New York, July 1964 -May 1970. 
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An additional search was made of the Research in Education in- 
dexes by computer in an attempt to identify material related to curr- 
icula development in the construction industries. 

Lewis, Wiley B. Revieu and Amlyaia of Curricula for Oooupations in 
Food Processing and Distribution, Information Series No. 32, 
ERIC Clearinghouse on Vocational and Technical Education, The 
Colter for Vocational and Technical Education, The Ohio State 
University, Columbus, Ohio, December 1970. (ED 045 820) 

Lewis, Wiley B. Reoien and Analysis of Curricula for Occupations in 
Health, Information Series No. 27, ERIC Clearinghouse on Voca,- 
tional and Technical Education, The Center for Vocational and 
Technical Education, The 0hi-» State University, Columbus, Ohio 
November 1970. (ED 044 507) 

Lewis, Wiley B. RevieiD and Analysis of Curricula for Occupations in 

Public Services, Informational Series No. 29. ERIC Clearinghouse 
on Vocational and Technical Education, The Center for Vocational 
and Technical Education, The Ohio State University, Columbus, 
Ohio, November 1970. (ED 045 813) 

Lewis, Wiley B. RevieD and Analysis of Curricula for Occupations in 
Transportation, Information Series No. 28, ERIC Clearinghouse 
on Vocational and Technical Education, The Center for Vocational 
and Technical Education, The Ohio State University, Columbus, 
Ohio, December 1970. (ED 045 814) 

Lumsdaine, A. A. and Glaser, Robert (eds.). Teaching Machines and 

Prograrmed Learning: A Source Book, National Education Associa- 
tion, Department of Audio-Visual Instruction, Washington, D.C., 
1960. 

McCord, Mary F. ERIC Products 1970-1971, An Annotated Bibliography of 
Information Analysis Publications of The ERIC Clearinghouses, 
July 1970 through June 1971, ERIC Clearinghouse on Library and 
Information Sciences, American Society for Information Science, 
Washington, D.C., Septanber 1971. 

Mathieson, David E. Correspondence Study: A Surmary Review of the 

Research and Development Literature, ERIC Clearinghouse on Adult 
Education, Syracuse, New York, March 1971. 

Beginning with a historical review of private correspondence 
schools, supervised high school level programs, military programs 
(notable the United States Armed Forces Institute), and activities 
by such university extension luminaries as William Rainey Harper and W. 



O 19 

ERIC 



H. Lighty, this literature review covers accreditation and licensing 
problems, general characteristics of students, educational methods and 
course design, and patterns of student methodology. Limitations 
affecting the effective use of correspondence study are assessed, 
along with the instructor role, steps in planning course content and 
materials, and the structuring of assignments and instructor response 
(feedback). Expected trends in the use of programmed instruction, 
broadcast media, films and other audio-visual resources, small groups, 
special degree programs, and arrangements for course credits and 
degrees through examinations, are also suggested. Annotated chapt nr 
bibliographies contain 164 referoices.—L. J. 

Mathieson, Noira B. A Bibliography of Bibliograpkiea on Career and 
Vocational Education^ Part II of the ERIC Clearinghouse on 
Teacher Education Project on Career Education, (SP 005 880), 
llashington, D.C., October 1972. (ED 067 387) 

Nesics, Emil A. Education and Training for Effective Manpower Utiliza- 
tion: An Annotated Bibliography on EdiiuxLtion and Training in 
Work Organizations, Bibliography Series No. 9, New York State 
School of Industrial and Labor Relations, Ithaca, New York, 
March 1969, 157 pp. 

The training function in organizations has grown in magnitude and 
status since its inception as & part of the formal structure in the 
early 1940. This bibliography is designed to cover the relevant and 
pertinent literature that has been concurrent with the growth and 
changed status of the training function. The major purpose of this 
volume is to serve the practitioner and student as a ready reference 
to the practices and theories on which concepts about learning at the 
workplace have emerged and developed. The references cited are drawn 
from the sparse literature, which appeared early in the century, to 
the alxindant and varied current writings in books and periodicals. 

The volume was assembled and organized so that the practitioner 
can find ready sources to help him cope with immediate day-to-day 
training problems. These problems may deal with the application of 
learning concepts and theories at the adult level across the wide 
work-force spectrum. Included are references to the handling of new 
employees in the process of initial orientation and job instruction 
as well as references to the retraining of older workers who are 
promoted to new jobs or who are, for one reason or another, trans- 
ferred to jobs requiring new knowledges and skills. Covered also 
are situations dealing with technician training and the handling of 
engineering skill obsolescence. The training specialist can also 
find references to a variety of methods and techniques for supervisor 
and manager development in his organization. 
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Naylor, J.C. and Briggu, G.E. Long-Term Retention of Learned Skills: 
A Review of the Literature^ Technical Report ASD-TR-61-390, 
August 1961. (AD 267 043) 

Pautler, Albert J. and Schaefer, Carl J. Reoiew and Synthesis of 

Research in Trade and Industrial Education, Information Series, 
ERIC Clearinghouse on Vocational and Technical Education, The 
Center for Vocational and Technical Education, The Ohio State 
University, Columbus, Ohio, September 1969. (ED 036 638) 

Price, R. G. and Hopkins, C. R. Review and Synthesis of Research in 
Business and Office Education (2nd ed.). Research Series No. 
55, ERIC Clearinghouse on Vocational and Technical Education, 
The Center for Vocational and Technical Education, The Ohio 
State University, Columbus, Ohio, April 1970. (ED 038 520) 

The review was, in general, limited to work completed during the 
period 1966-1968. In order to identify research completed during that 
period, the reviewers searched the usual library sources. Another 
phase of the search was a mail survey of representatives of the Nation- 
al Association for Business Teacher Education, state supervisors, and 
teacher educators. 

Because of the need to include only those studies relating to 
the topics covered in the review, efforts were made to screen the 
report by: 1) title, 2) abstract, and 3) primary source. As a result 
of this screening, some pertinent reports may have been overlooked. 

Research studies included are primarily those that are relevant 
to business and office preparation at senior high schools and two- 
year post -secondary schools. Also included are reports pertaining 
to business teacher education. — Auth. 

Rose, A.J. and Turner, T.B. Skill Loss: An Assessment of Eyaluaiion 
Techniques Used by Other Services and Their AippUcatvon to 
Navy Technical Ratings, PRL Report No. WRM 67-24 Naval Personal 
Research Laboratory, Washington, D.C., January 1967. (AD 649 554) 

This report represents the initial phase of research in the 
design and development of techniques to reduce loss of technical 
skills caused by sustained non practice of the skills. The greatest 
incidence of skill loss in the Navy is presently considered to occur 
when personnel are assigned to non-rating related billets ashore. 

This report is primarily a bibliographic survey of research 
previous! V or currently conducted by the Navy and other military 
services.' Conclusions drawn from this survey are: 1) most of this 
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research concerns operating rather than maintenance skills and has to 
do with perfecting initial training methods that will prolong job- 
skill retention, 2) specific research in progress is mainly oriented 
toward the long-term memory of motor skills or complex, interacting 
team skills on integrated systems, and 3) there is a conspicuous lack 
of research related to the loss of skills caused by non-practice to 
those skills. 

The report recomuierids that subsequent research be directed to- 
ward: 1) Identifying ratings and skills levels of ratings in which 
loss presently occurs, 2) determining the extent of skill loss en- 
countered, and 3) determining in each instance specific courses of 
action designed to enhance rating skill retention.—Auth. 

Schultz, D.G. and Siegel, A.I. Post-Training Performance Criterion 
Development and Application: A .Selective Review of Methods 
for Measuring Individual Differences in On-the-job Performance, 
Applied Psychological Services, Ifayne, Pa., July 1961. 

Sivatko, John R. ^ ••Correspondence in5:ruction,** in Ebel, Rob^t (ed.). 
Encyclopedia of Educational Research (Ath ed.), Macmillan, New 
York, 1969, pp. 213-218. 

Snith, Robert G., Jr. An Annotated Bibliography on the Design of 

Instructional Systems, Technical Report 67-S, The George Wash- 
ington University, Human Resources Research Office, Alexandria, 
Va., May 1967. 

Smivh, Robert G., Jr. An Annotated Bibliography on the Determination 
of Training Objectives, Research Memorandum, The George Wash- 
ington University, Alexandria, Va., June 1964. 

Snyder, T.R. and Butler, R.L. Review and Analysis of Curricula for 
Occupations in Metalworking, Information Series No. 25, ERIC 
Clearinghouse on Vocational and Technical Education, The Center 
for Vocational and Technical Education, The Ohio State Univer- 
sity, Columbus, Ohio, October 1970. (ED 044 4 95) 

Tennyson, Robert D. "A Review of Experimental Methodology in Instruc- 
tional Task Sequencing. »» AV Communication Review, Vol. 20, No. 
2, Summer 1972, pp 147-159. 

Twelker, P.A., Urbach, F.D., and Buck, J.E. (U.S. International Univer- 
sity in Oregon, Corwalls, Oregon), Tne Systematic Development of 
Instruction: An Overview and Basic Guide f-c the Literature, 
ERIC Clearinghouse on Media and Technolog), Stanford, Calif., 
March 1972. 
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Ugelow, A. Motivation and the Automation of Training: A Literature 
Revieu, Technical Documentary Report M.iL-TDR*62-15, March 1962. 
(AD 277-287) 

U.S. Department of Health, Education, and Welfare, Public Health 
Service. Training Methodology^ Bart III: ^ Instruotional 
Methods and Techniques; an Annotated Bibliography , Public 
Service Publication 1962, Part III, National Communicable 
Disease Center Training Program, Washington, D.C. 1969. 

U.S. Department of Health, Education, and Welfare, Public Health 

Service. Training Methodology. Part IV: Audiovisual Theory, 
Aide and Equipment; an Annotated Bibliography^ Public Health 
Ser^^ice Publication 1962, Part IV, National Communicable 
Disease Center Training Program, Washington, D.C, 1969. 

U.S. Department of the Air Force. Air Force Human Resources Labora- 
tory, Personnel K-^search Division. Abstracts of Technical 
Reports, 1969'19?1 TSupplement to Abstracts of Personnel Re- 
search Reports: VIII. 1954-1968), Uckland AFB, Texas, Decem- 
ber, 1971. 

Includes abstracts of 48 technical reports issued by Personnel 
Research Division from January 1969 through December 1971.— ff.ir. 

U.S. Department of the Army. Planning and Evaluation of Personnel 

Management Programs, A Bibliographic Surrey, DA Pamphlet 600-9, 
Washington, D.C, 1 September 1969. 

U.S. DeparMnent of Labor, Library. Cost-Benefit Analysis: Theory and 
Application to Manpower Training Programs; A Bibliography, 
Current Bibliography No. 3, Washington, D.C, May 1971, 44 p. 
(PB 202 158) 

U.S. Department of Ubor, Library. Program and Tx^ining Evaluation: 
Selected References, Washington, D.C, April 1973* 

U.S- Department of the Navy. Bibliography and Abstract of Technical 
Reports July 1969 to June 1970, WRR 71-2, Naval Personnel Re« 
search and Development Laboratory, Washington Navy Yard, Wash- 
ington, D.C, October 1970. 

Valverde, Horace H. and Youngs, Eleanor J. Annotated Bibliography of 
the Training Research Division Reports (19Su-1969), Technical 
Report AFHRL-TR-69-11, Air Force Human Resources Laboratory, 
Wright-Patterson Air Force Base, Ohio, September 1969.AD-865 713. 
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Annotated Bibliography of technical reports and other publica- 
tions (memo reports, journal articles) on research conducted by the 
Training Research Division covering period July 1950 through July 
1960."ff.ff. 

Valverde, Horace H. K^intenanoe Training Media: An Annotated Bib-- 
liography. Technical Report AMRL-TR-67-151, May 1968, 
(AD 673 371) 

Warmbrod, Catharine P. Review and Synthesis of Literature on Reai- 
dential Schools in Vocational and Technical Education^ Infor- 
mation Series, ERIC Clearinghouse on Vocational and Technical 
Education, The Center for Vocational and Technical Education, 
The Ohio State University, Columbus, Ohio. December 1970. 
(ED 045 821) 

This Review and Synthesis of Research on Cooperative Vocational 
Education is one of a series of "State-of-the-art" papers in voca- 
tional and technical education and related fields. It should assist 
in identifying substantive problems and methodological approaches foi" 
researchers and curriculum development specialists, as well as pro- 
viding practitioners with a summary of research findings that have 
application to educational programs. In the field of vocational 
and tectoical education, the pace of research and development activ- 
ities ?ias increased considerably during the period under review. 

As one of a series of information analysis papers released by 
the ERIC Clearinghouse on Vocational and Technical Education, this 
review is intended to provide researchers, curriculum development 
specialists, and practitioners with an authoritative analysis of the 
literature in the field. Those who wish to examine primary sources 
of information should utilize the bibliography. Where ERIC Document 
numbers and ERIC Document Reproduction Service prices are cited, 
the documents are available in microfiche and hard copy forms. — Auth. 

Weisgerber, R. A. (ed.). Perspectives in Individualized Learning^ 
F. F. Peacock Publishers, Inc., Itasca, 111., 1971, 406 pp. 

The readings presented are an analysis of selected factors 
underlying the process of individualized learning. The book is organ- 
ized topically and moves from theoretical considerations toward an 
analysis of important educational components. The readings come from 
a cross section of experts representing the areas of learning theory, 
individual differences, measurements and evaluation, educational 
objectives, teacher roles, learning activities, facilities technology, 
and computer systems. Each chapter is prefaced with an introduction 
by the editor. Chapter topics include: underlying assumptions 
concerning the need for individualized learning, mental abilities as 
a possible basis for individualization, the impact of individual 
differences on reading, the measurement and accommodation of individual 
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differences, educational objectives, evaluation, the changing role 
of the teacher, individualized and interactive learning activities, 
the instructional environment, and computer -assisted instruction. 
— Auth, 

Weinrich, Ralph C. Review and Synthesis of Research on the Administra- 
tion of Vocational and Technical Education, Information SCiies, 
ERIC Clearinghouse on Vocational and Technical Education, The 
C«iter for Vocational and Technical Education, The Ohio State 
University, Columbus, Ohio, March 1970 (ED 037 542) 

Wbod, W.D. and Campbell, H.P. Cost-Benefit Analysis and the Economics 
of Inoestment in Human Resources. An Annotated Bibliography, 
Bibliography Series ilo. 5, Industrial Relations Centre, (>ieen's 
University, Kingston, Ontario, Canada, 1970, 217 pp. (ED 045 848) 

This annotated bibliography presents 389 citations of periodical 
articles, monographs, and books, and represents a survey of the lit- 
erature as related to the theory and application of cost-benefit an- 
alysis. Listings are arranged alpnabetically in these eight sections: 
(1) Human Capital, (2) Theory and Application of Cost-Benefit Analysis, 
(3) Theoretical Problems in Measuring Bene. its and Costs, (4) Invest- 
ment Criteria and the Social Discount Rate, (5) Schooling, (6) Train- 
ing, Retraining and Mobility, (7) Health, and (8) Poverty and Social 
Welfare. Individual aitries include author, title, source informa- 
tion, "headings" of the various sections of the document which provide 
a brief indication of content, and an annotation. This bibliography 
has been designed to serve as an analytical reference for both the 
academic scholar and the policy-maker in this axea..—Auth. 
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SECTION IV 



HANDBOOKS AND MANUALS 



Generalized "how-to-do-it*' handbooks and manuals have been 
developed fur OJT programs in many different organizations. There 
are also specialized manuals concerned with specific techniques. 

The citations in this section are of interest to those who 
have responsibility for designing new manuals for specific organi- 
zations as well as to those who are looking for guidance regarding 
how to approach specific problems in OJT. 



BIBLIOGRAPHY 



Allen, D., Hahn, B. J., Johnson, M. P., and Nelson, R. S. Foly- 

sensory L earning Hfirongh Multi-Media Instruction in Trade and 
Taohnioal Education, Division of Vocational Education, Univer- 
sity of California, and Bureau of Industrial Education, 
California State Department of Education, Los Angeles, Calif., 
1968. (ED 022458) 

This report explains a teaching system designed to stimulate poly- 
sensory learning by use of multimedia instructional materials, which 
use as many of the physical senses as practical to augment traditional 
instruction. They include motion pictures, filmstrips, audio tapes, 
models, mock-ups, etc., according to school facilities and course 
needs. Care should be used in buying such expensive media as 8mm 
sound films, since technical instruction needs constant updating; 
tapes and filmstrips are low in cost and easy to produce. The system 
is planned for use by each student at a study booth providing privacy 
and protection from distraction. The student may use the material at 
his own pace, before or after a shop, laboratory, or small-group dis- 
cussion. This system is expected to stimulate motor skills, cognitive 
learning, and the concomitant attitudes of appreciation, responsi- 
bility, etc, appropriate to the student's maturity. This individ- 
ualized instruction and traditional teaching are compared in detail by 
objective, technique, work method, evaluation, and the role of ad- 
visory committees, supervisors, or administrators. Ideally the 
multimedia method is so organized and used that the program is system- 
atically evaluated and revised to meet realistic student performance 
goals efficiently. Details of the system, examples of its use, 
methods of preparing the materials, evaluation of its results, and 
its future possibilities are presented. 
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Barber, J, W. (ed.)- Industrial Training Handbook^ A. S. Barnes and 
Company, New York, 1969, 

This book may^ be used with the related six filmstrip series, "ON-THE- 
JOB TRAINING" and leader's guide, also available from Addison- 
Wesley — Auth. 

Bolton, Dale L. l^aher EvaLmtiorij PREP Report No. 21, (Putting 

Research Into Educational Practice Series), U. S. Department of 
Health, Education, and Weitare, Office of Education, National 
Center for Educational Comnrunication, DHEW Publication No. (OE) 
72-9, Washington, D, C, 1972. 

All teachers are evaluated. Regardless of how formal the system for 
evaluation is, what evidence is collected or analyzed, how often fr^al 
reports are written--teachers are evaluated and they are evaluateo 
rather often. Students, parents, other teachers, administrators 
supervisors, and even the public evaluate teachers. The questio 
facing both administrators and teachers, then, is not whether tt jlS 
should be evaluated, since this cannot be avoided; rather the qu^^cxon 
is how systematic the evaluation should be in order to be most effec- 
tive. 

Effectiveness must be in terms of certain purposes desired for the 
school district; and the design of the evaluation system should in- 
clude ways of collecting and processing information, communicating with 
the people concerned, making decisions, and assessing how well the 
evaluation system works. --ylwth. 

Briggs, L. J. Handbook of Procedures for the Design of Instruction, 
Monograph 4, American Institutes for Research, Pittsburgh, 1970. 

Briggs, L. J., Carapeau, P. L., Gagn6, R. M-, and May, M, A. In- 

structional Media: A Procedure for the Design of Multi-media 
Instruction^ a Critical Review of Research and Suggestions for 
Future Research, Monograph 2, American Institutes for Research, 
Pittsburgh, 1967. 

Broadwell, Martin M. Hhe Supervisor and On-the-Job Training, 
Addis on- Wesley, Reading, Mass., 1969. 

Here's an easy-to-undersnand approach to on-the-job training for the 
individual supervisor. The book is a suitable handbook for trainers 
on how to do OJT. 

Using a systematic, but nontechnical approach, chapters cover Why 
Train; How People Learn: OJT or Classroom Training?; Analyzing the 
Job; Determining Objectives; Preparing to Train; How to do OJT; and 
Pre- and Post-training Evaluation. The book will help the supervisor 
to analyze the job, prepare himsc^lf and the employee, proceed through 
the steps, and provide follow-up. 
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Cenci, Louis. Skill Training for the Job^ Pitman Publishi 
New York, 1966. 



Chenzoff, Andrew P. and Folley, John D., Jr. Guidelinea for Training 
Svtuatyan AnalyBie (EA), Technical Report NAVTRADEVCEN 1218-4, 
Prepared by Applied Science Associates, Inc., Valencia, Pa., for 
U. S. Naval Training Device Center, Port Washington, New York. 
July 1965. 

These guidelines represent a textbook for instruction in three phases 
of Training Situation Analysis (TSA) , a standardized procedure, devel- 
oped by NTDC, for systematically gathering and interpreting the in- 
formation that is relevant to the planning of training and training 
devices. 



Three-phases of TSA are described in detail: System Familiarization, 
Task Analysis Method (TAM) and Training Analysis Procedure (TAP). 
System FamiliarizatioT/ provides an orientation to the training problem, 
the system structure and flow, and the equipment. Task Analysis Method 
produces a set of task descriptions containing the information neces- 
sary for making training device decisions. Training Analysis Procedure 
produces a ranking of tasks based upon the potential benefit to system 
performance as a result of training and the cost of that training. 
Recommendations for the conduct of these three phases and suggested 
working forms are presented. 

Craig, R. L. and Bittel, L. R. (ed.). Training and Deo elopment hand- 
book, McGraw-Hill, New York, 1967. 

Department of Employment and Productivity. Ihe Training and Use of 
Operators ae Instructore, Her Majesty's Stationery Office, 
London, 1969. 

Folley, John D., Jr., Joyce, Reid P., Mallory, William J. (Applied 

Science Associates, Inc.), and Thomas, Donald L. (Air Force Human 
Resources Laboratory). Fully Prooeduralized Job Performance Aids: 
Volme I-Draft Speoifioation for Organizational Maintenance, 
Technical Report AFHRL-TR-71-53, Volume I, Air Force Human Re- 

A^Hrt/^no^i^^"'^^^^'^' Wright-Patterson AFB, Ohio, December 1971. 
AD-70h 903. 

This report supplies a model for specifications for the preparation of 
Fully Procedural i zed Job Performance Aids for the operational mainte- 
nance of Air Force man-machine system. The model reflects the research 
findings of AFHRL and other DOD agencies concerning maintenance data. 
It has the unique feature of requiring that certain subproducts neces- 
sary for the development of this type of data be prepared in a stan- 
dard format and submitted for review by the procuring agency. These 
subproducts include items such as a task identification matrix, task 
inventory, a task description index and management matrix, and task 
step data details. The aids to be developed from these specifications 
are for the organizational maintenance of any man-machine system and 
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support the performance of the following maintenance functions: Check- 
out, alignment, repair, adjustment, calibration, malfunction isolation, 
and the removal and replacement of malfunctioning equipment items. It 
calls for the preparation of the aids in several options of job guide 
format .--i4wt/z. 

Folley, John D., Jr., Joyce, Reid P., Mallory, William J. (Applied 

Science Associates, Inc.), and Thomas, Donald L. (Air Force Human 
Resources Laboratory). Fully Prooedumlized *Tdb Performance Aids* 
Volume II"Developer'8 Handbook^ Technical eport afhrL-TR-71-53, 
Volume II, Air Force Human Resources Laboratory, Brooks, AFB, 
Texas, December 1971. AD~744 007. 

This report provides guidance for the development of fully proceduraliz- 
ed job performance aids for the organizational maintenance of Air Force 
man-machine systems. It contains detailed instructions for preparing 
fully procedural i zed job performance aids in accordance with the re- 
quirements of the draft specification contained in AFHRL-TR-71-53(I) . 
It includes instructions for performing the behavioral task analysis 
and for converting the results of the analysis into effective perfor- 
mance aids. In addition, it presents a strategy and guidance for 
developing supervised practice exercises designed to produce the skills 
required to prepare fully proceduralized job performance aids. — Auth. 

Cropper. G. L. A Taohnology for Developing Imtruotional Materials, 
Vcl.l. User's Manual; Vol. 2. Orientation^ Vol. 3. Handbook; 
Vol. 5. Final Exercises, American Institutes for Research, 
Pittsburgh, September 1971. 

Haggard, Donald P., Willard, Norman, Jr., Baker, Robert A., Osborn, 
William C, and Schwartz, Shepard. An Experimental Program of 
Instruction on the Management of Training, Technical Report 70-9, 
Human Resources Research Organization, HumRRO Division No. 2, 
Fort Knox, Kentucky, June 1970 (AD 711 948) 

A 96-hour course on the management of training was developed and 
evaluated for presentation to advanced officer classes. The general 
procedures followed in developing the course were construction based 
on job task statements, a systems-engineering approach to training, 
and state-of-the-art technology; repeated presentation of course 
materials, and modification by training research personnel on the basis 
of student and instructor appraisals. The course was then presented 
and evaluated by a military instructor. Critiques by military students 
and staff indicated that the course had considerable value, but re- 
quired some additional revision before adoption by the Army. Suggested 
revisions were: (a) reducing course length, (h) integrating the course 
and CONARC Regulation 350-100-1, (c) coordinating supporting literature 
and course objectives, and (d) implementing the revised course as a 
common subject in officer training programs. These revisions would not 
require further research and development and could be accomplished by 
military operational training personnel with technical advice by re- 
search personnel. 



Hummel, Lester F. and Nevnnaster, Ronald Computerized PreparaHon of 
Navy !Er-aining Manuals^ Report 95, Operations Analysis Department, 
Navy Fleet Material Support Office, Mechanicsburg, Pa., 17 January 
1973. (AD 754 431) ^ 

Navy training manuals require continuing revision in order to remain 
current. Considerable time and man-effott are required to accomplish 
the revision of such manuals. The use of a computer assisted document 
preparation system in the update of these manuals offers potential for 
significant reductions in both the elapsed time and the man-effort re- 
quired to complete a revision pycle. This report relates the results 
of a pilot project initiated to test the feasibility of using such a 
computer-based text-editing system to update Navy training manuals. 
The Computerized Specification Management System (CSMS), a text-editing 
system developed by Naval Electronics Uboratory Center (NELC), San 
Diego, California, was used to prepare revised editions of two Navy 
training manual s and their associated correspondence courses. Modifi- 
cation and enhancement of the original system was required to provide 
additional font and character capabilities. Significant reductions in 
elapsed time and man-effort required to update a manual were achieved. 
Additional texts will be assigned for update via the CSMS system and 
efforts to refine and improve the system will continue. 

Jacobs, Paul I., Maier, Milton H., and Stolurow, Lawrence M. A Guide 
to Evaluating Self-^instructional Programs, Holt, Rinehart 5 
Winston, 1966. 

Joyce, Reid P., Folley, John D., Jr., and Elliott, Thomas K. (Applied 
Science Associates, Inc.). Fully Proaeduralized Job Performance 
Aids. Volume III~JPA Manager's Handbook, Technical Report AFHRL- 
TR-7J-53, Volume III, Air Force Human Resources Laboratory, Brooks 
AFB, Texas, December 1971. AD- 744 817. 

This report provides guidance for the Air Force data managers charged 
with the responsibility for the procurement of fully proceduralized 
JPAs. It provides guidelines, suggested procedures, and checklists for 
use by data managers in the review and assessment of the subproducts, 
intermediate products and JPAs produced in accordance with the draft 
specification contained in AFHRL-TR-71-53(I). 

Mager, Robert F. and Beach, Kenneth M., Jr. Developing Vocational 
Instruction, Fearon Publishers, Belmont, Calif., 1967. 

Developirq Vocational Instruction describes the steps involved in pre- 
paring instruction that can be demonstrated to facilitate learning. In 
so doing, the concern is not with what would be the easiest thing to 
do, but what will be the most professional. Probably better than any- 
one else, vocational and technical instructors realize that there can 
be just as big a difference between practicing a skill and teaching it 
as there is between teaching and telling. The goal of this book is to 
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describe the steps that must be carried out if one is to become as 
expert in the skill of systematic course development as he is i.. the 
practice of his own vocation or technical specialty. 

The procedure of systematic course development outlined in this book is 
not specific to subject matter or vocation, and it applies to many 
academic as well as vocational and technical areas • — Auth. 

Margolis, Fredrick H. Tpaining by Objeotivee: A Participant-oriented 
Approach, Behavioral Science Center of Sterling Institute, 
Washington, D. C, June 1970. 

Miller, R. B. Handbook on Inining and Training Equipment, Technical 
Report WADC-TR-53-136, Aero Medical Laboratory, Wright Air Devel- 
opment Center, Wright-Patterson AFB, Ohio, Jur.e 1953- (AD 16 859) 

This Handbook is intended to aid in preparing recommendations on the 
design and use of training equipment. As such, it permits cross refer- 
encing to a companion report, WADC Technical Report 53-138, Rumcax 
Engineering Design Schedule for Training Equipment. The contents of 
the Handbook include learning and transfer theory, principles applicable 
to problems of training, and bibliographic references. One principal 
theme that is developed is that different kinds of tasks, and different 
degrees of learning have different implications for transfer of train- 
ing and the best form of presenting knowledge of results. Stages of 
learning are analyzed in detail, as are the variables in "knowledge of 
results." The principal sections are titled: I. Human Learning as 
An Overview; II. The Role of the Instructor in Training; III. The 
Trainer as a Demonstrator of Principles; IV. The Use of Knowledge of 
Results; V. The Problem of Simulation; VI. The Problem of Motivation; 
VII. Preparing the Specifications for a Training Device. The many 
problms indicated as requiring further research in the field of human 
learning and training suggest the importance of "programmatic" studies. 
These materials, in conjunction with WADC TR 53-137, A Method for tian- 
Machine Hhsk Analysis, and WADS TR 53-135, A Method for Determining 
Human Engineering Design Requirements for Training Equipment, are 
designed for use by trained personnel. — Auth. 

McGehee, W. and Thayer, P. W. Training in Business and Industry, John 
Wiley and Sons, New York, 1961. 

Here is a book that places training in perspective as a mangement tool 
for achieving organizational goals in business and industry. Instead 
of merely surveying available training techniques, it critically 
examines them on the basis of recent research and actual business and 
industrial experience. The aim of the book is to develop a sound 
rationale for analyzing and solving the training problems encountered 
in practice. 
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All four basic elements involved in the development of a training 
program are considered in detail: determining training needs; utiliz- 
ing data dravm from the psychology of learning; selecting appropriate 
techniques; and assessing training results. Careful consideration is 
also given to the allocation of responsibility for training in industry. 
Emphasis throughout is placed on harmonizing training programs with 
clearly defined goals and on using learning theory and research as 
bases for choosing training procedures. Particular attention is given 
to such important recent developments as teaching machines, business 
games, and sensitivity training. An extensive bibliography is in- 
cluded. ~i4Mt?z. 

National University Extension Association. Educational T&ohnology , 
prepared by National University Extension Association, Silver 
Spring, Md. (under a grant from the U. S. Office of Education), 
June 1968. 

Ofiesh, G. D» Programed Instruction: A Guide for Management^ American 
Management Association, New York, 1965. 

Training is a costly business. Some have estimated its cost to industry 
alone as $30 billion annually. But all too often it is wasted. . .ill- 
conceived, and poorly managed. 

The past few years have witnessed a sharp increase in industry's use of 
programed instruction to correct this inefficiency. Its effectiveness 
as a training technique has been proved in factories and offices across 
the nation... its value as a management tool has fast become an accepted 
fact. This book analyzes the emergence of programed instruction, ex- 
plains the benefits of a programed- instruction-based training system, 
and discusses its applications to many industrial training problems. 

This report contains case histories that show how this new training 
technique is being used in General Electric Company, International 
Business Machines Corporation, Trans World Airways, American Telephone 
and Telegraph Company, Burroughs Corporation, Liberty Mutual Insurance 
Company, Pfizer Laboratories, and numerous other industries, associa- 
tions, and educational institutions. These examples clearly demonstrate 
the potential of programed instruction and point the way to new and 
greater training achievements. ~i4wt?i. 

Parker, J. F., Jr. and Downs, J. E. Selection of Training Media j 

Technical Report ASD-TR-61-473, Behavioral Sciences Laboratory, 
Aerospace Medical Laboratory, Aeronautical Systems Division, 
Wright-Patterson AFB, Ohio, September 1961. (AD 271-483) 

The selection of training media in support of military training programs 
represents an important area of concern. Training equipment exercises 
considerable influence on the way in which training programs are con- 
ducted, upon their effectiveness in accomplishing objectives, and upon 
the total cost of the program. 
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This report is designed to assist a training analyst faced with the 
problem of selecting specific training aids and devices to be used in 
support of the development of the personnel subsystem of a military 
system. The translation of statements of desired personnel performance 
and capabilities, as presented in Qualitative and Quantitative Person- 
nel Requirements Information and task analysis documents, into training 
objectives is discussed. The effectiveness of various training media 
in meeting specific training objectives is indicated and justified in 
terms of available objective evidence. An example is presented illus- 
trating the manner in which training media are selected in support of a 
typical Air Force operator position. — Auth. 

Popham, W. J. and Baker, E. L. Establishing Instruotional Goals j 
Prentice-Hall, Englewood Cliffs, N. J., 1970. 

This book consists of a collection of five self-instruction programs 
designed to be completed individually by the reader. The programs deal 
with various aspects of instruction and are intended to provide a set 
of tangible competencies that can be employed by a teacher in making 
instructional decisions. The focus of the programs in this volume is 
on the topic of instructional goals: how to select them, how to state 
them, and how to establish pupil performance standards for such goals. 
After completing the five programs, you will be skilled in dealing with 
questions related to educational objectives. The competencies provided 
by the programs should be of considerable value to individuals who are 
preparing for a teaching career at any level of instruction, kinder- 
garten through college. Experienced teachers will also find that the 
topics treated in the programs bear upon many practical decisions that 
they must make regarding their instruction. In essence, then, both 
preservice and inservice teachers should profit from completing the 
programs contained herein. — Auth. 

Rundquist, Edward A. "Extension, Facilitation, and Validation of a 
Course Design procedure and its Basic Concepts," (Project No. 
F39522, Work Unit-0162), Naval Personnel and Training Research 
Laboratory, San Diego, Calif. (In preparation, June 1971). 

In general, training courses are not designed to be so effective 
and efficient as they could be. Previously an integrated procedure 
for the design and improvement of school training courses was developed. 
When the procedure is applied under professional guidance, marked 
gains in training occur. Certain of the steps, notably the task 
analysis, have been found very difficult for typical Navy instructors 
to apply. It is one of the purposes of this work unit to discuss ways 
of simplifying the application of such steps. Two additional purposes 
are to: (1) determine whether the procedure can be extended to more 
complex duty assignments and to shipboard training; and (2) validate 
further selected course design and training concepts basic to the 
application of the procedure. — Auth. 
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Rundquist, Edward A, Job Tpairdng Course Desigyt and Improvement (2nd 
ed.), Researcii Report SRR 71-4, Naval Personnel and Training Re- 
search Laboratory, San Diego, Calif., September 1970, (AD 876 204) 

The second edition of the course design manual is a thorough re* 
vision of earlier editions • The manual is designed to assist instruc- 
tors in developing and improving job-related training courses. Major 
changes from earlier editions include more careful definitions of 
training and training related terms; a general clarification of con- 
cepts and procedures, especially those concerned with job and skill 
analysis for training purposes; more emphasis on principles of devel- 
oping training exercises; a more thorough consideration of the im- 
portance and means of adapting individual differences; and more 
emphasis on the significance of the course mission for course design. 
Examples from a wide variety of duty assignments are included. ~/lwt?z. 

Smith, Robert G., Jr. Controlling the Quality of Training^ Technical 
Report 65-6, The George Washington University, Human Resources 
Research Office, Alexandria> Va., June 1965. (AD 618 737) 

The need for a quality control system in a military training 
program and methods of establishing such a unit are described and 
evaluated in this report, which is part of a research project in the 
technology for developing training. It is stated that the purpose of 
quality control is to ensure a satisfactory standard of competence 
among the students who graduate, to maintain this quality by a contin- 
uous monitoring process, and to improve training where it is found to 
be deficient. In order to function successfully, a quality control 
system should constitute a separate unit, independent of, but coop- 
erating with, the instructional departments. Attention is given to 
proficiency testing as the chief means of measuring the success of the 
training program, with emphasis upon the importance of a uniform 
standard and consistent method in the preparation, administration, and 
scoring of tests .—Auth. 

Smith, Robert G., Jr. The Design of Instructional Systems^ Technical 
Report 66-18, The George Washington University, Human Resources 
Research Office, Alexandria, Va., November 1966. 

This report, based on an extensive survey of current literature, 
describes a system approach to designing training and considers factors 
bearing on training effectiveness. An efficient instructional system 
is conceived as one in which the components form an integrated whole, 
achieving maximum effectiveness at the least possible cost. Components 
considered in this report include presentation media, student manage- 
ment, techniques for practicing knowledge and performance, knowledge 
of results, directing student activities toward the goals of the train- 
ing program, and testing and evaluating the system in terms of 
efficiency and cost. — Auth. 
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Smith, Robert G., Jr. Ihe Development of Tmining Ohgeotives, Research 
Bulletin 11, The George Washington University, Human Resources 
Research Office. Alexandria, Va,, June 1964- 

This bulletin describes modem concepts and techniques used in deter- 
mining training objectives, A training objective is a precise clear 
statement of one of the performances expected of a student upon 
completion of a course, a complete list of such objectives constituting 
the* mission of a course. The concepts and techniques described are 
based on research performed by both military and civilian researchers 
in the field of training. 

The development of job-related, detailed statements of objectives is a 
matter of the first importance in designing effective training programs • 
These objectives permit every element of a training program— lessons, 
texts, practical exercises, and examinations--to fall into line in a 
consistent manner. Training objectives that are job relevant will pro- 
vide for a course that wiii also be responsive to the needs of the unit 
to which the soldier will go after training, and that includes little 
irrelevant content. 

The techniques described have been selected as being practical for 
Army training personnel. Those who wish to learn more of the theory 
and' techniques of determining objectives will find a selected bibliog- 
raphy listed by chapter at the end of the bulletin. Tne rationales for 
the various aspects of developing objectives will be presented, and 
appropriate choices of techniques indicated. — Auth. 

Smith, Robert G., Jr. The Engineering of Educational and Tmining 
Systems^ D. C. Heath and Co., Lexington, Mass., 1971. 

In writing this book on the engineering of education and training 
systems, I have tried to explain the implications of systems ideas for 
education and training and to provide sufficient descriptions of suit- 
able methods and techniques, so that either with this book alone, or 
in a few cases, by reterence to cited works of others, the reader could 
design and implement effective systems. 

I have resisted the temptation to be highly critical of modem practices 
in education and training. Where critical comment is made, it has been 
inserted primarily to clarify an example. Following the same theme, 
the final%hapter presents a series of positive suggestions for dealing 
with what is obviously a current educational crisis of major dimensions. 
—Auth. 

Stokes, P. M. Total Job Training: A Manual for the Working Manager ^ 
American Management Association, New York, 1^66. 
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Training Psychology Branch, Behavioral Sciences Laboratory, Aerospace 
Medical Laboratory. Uses of Task Analysis in Deriving Training 
and Training Equipment RequirementSj Technical Report WADD-TR- 
60-593, Wright Air Development Division, Air Research and 
Development Command, Wright-Patterson AFB, Ohio, Deceniier I960. 

The requiremt'^ts for and uses of task information in developing re- 
quirements foi training eo^uipment are discussed in a series of seven 
papers by the human factors subcontractors involved in the development 
of three coBq[>lex electronic reconnaissance systems. The papers deal 
with the purposes, content, sources, and recording of task information. 
They cover the uses of task analysis infoirmation in establishing 
training requirements, selecting training equipment, and developing 
proficiency measures. A final paper describes in detail the approach 
taken on these issues with the AN/ULD-1 system. — Auth. 

U. S. Civil Service Commission, Bureau of Training. Training Eval- 
tiation: A guide to its Planning^ Development^ and Use in Agency 
Training Courses^ Training Systems and Technology Series No. 4> 
Pamirfilet T-13, U. S. Government Printing Office, Washington, D. C, 
May 1971. 

U. S. Department of Health, Education, and Welfare, Office of Educa- 
tion, Bureau of Adult, Vocational, and Technical Education. 
Motivation and the iHsadvantaged Trainee: A Manual for In-- 
structors^ (OE 37068) U. S. Government Printing Office, Washington, 
D. C, 1970. 

U. S. Department of Labor, Manpower Administration, A Handbook for 

Upgrading LotJ^Skilled Workers, Research and Demonstration Findings 
No. 13, prepared by Humanic Des5.gns Corporation, for U. S. Depart- 
ment of Labor, U. S. Government Printing Office, Washington, 
D. C, 1971. 

This handbook for upgrading low- ski 11 workers attempts to synthesize 
the experience and findings of more than four years of research and 
demonstration work in upgrading under-employed workers in the plant 
environment. It contains procedural and methodological guidelines for 
individuals and organizations that are undertaking in-plant programs 
to upgrade low-skill workers to higher levels of work and productivity. 
The guidelines are not intended to be job or program specific, but 
rather apply to a wide range ^f industry conditions and problems for 
which in-plant upgrading programs provide at least a partial solution. 
--Auth. 

U. S. Department of Labor, Manpower Administration. Touard the Ideal 
Joumeyma/t Vol^Mne 2. The Training System in the Pipe TxKzdes, 
Manpower Research Monograpi* jJo. 20, U. S. Government Printing 
Office, Washingtan, D. C, iJ71. 
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U. S. Department of Labor, Manpower Administration. Ibward the Ideal 
Jommeyman. Volume 3. Apprenticeship Tmining in the Machinist 
and Ibol and Die Maker Trades^ Manpower Research Monograph No. 20, 
U. S. Government Printing Office, Washington, D. C, 1971. 

U. S. Department of Labor, Manpower Administration. Toward the Ideal 
Joiameyman. Volume 4. The Training System in the Printing 
TradeSy Manpower Research Monograph No. 20, Manpower Adminis- 
tration, Washington, D- C, 1971. 

U. S. Department of the Navy, Bureau of Naval Personnel. All Hands 
(Nav-Pers-0), Number 611, Washington, D. C, December 1967. 

Warmbrod, J. Robert. Review and Synthesis of Research on the 

Economics of Vocational Education^ Information Series, The 
Center for Vocational and Technical Education, ERIC Clearinghouse 
on Vocational and Technical Education, Th ^hio State University, 
Columbus, Ohio, November 1968. (ED 023 9o. 

Warren, M. W. Training for Results: A systems Apporach to the 
Development of Human Resources in Industry, Addison- Wesley, 
Reading, M, Mass., 1969. 

Weisgerber, Robert A, (ed.) Institutional Process and Media Innovation, 
Rand-McNally, Chicago, 1968. 

Willis, M. Paul and Peterson, Richard 0. (American Institutes for 
Research). Deriving Training Device Implications From Learning 
Theory Principles • Volume I: Guidelines for Training Device 
Design^ Development and Use, Technical Report NAVTRADEVCEN 784-1, 
U. S. Naval Training Device Center, Port Washington, New York, 
July 1961. 
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SF.CTION V 



TRAINING COST-EFFECTIVENESS LITERATURE 



From a classical, economical point of view (e.g.. Mincer, 1962; 
Becker, 1962; and Canby, 1972), vocational training is an investment by 
the person as well as the organization. A number of mathematical for- 
mulas and several strategies for optimizing one's return on such invest- 
ments have been developed (e.g., Bateman, 1966), Most of these for- 
mulas, complicated as they may be, are woefully inadequate for two rea- 
sons. They are almost never based upon data; and they do not consider 
all the factors that need to be considered. 

There is general agreement, for example, that the following factors 
should be considered in cost-effectiveness algorithms about OJT: 
(a) the possibility of selectively rect. iting people in certain fields 
as an alternative to training inexperienced civilians (Hanushek, 1971; 
Kim, Lieser, Scheirer, and Minckler, 1968J; (b) the differential re- 
lationship of OJT and resident school training to unit readiness 
(O'Flaherty and O'Rorke, 1967); (c) the impact of the value of military 
training in civilian careers upon leaving the service (McCall and 
Wallace, 1967); (d) the impact of the military organization's reputa- 
tion for training upon the enlistment rate (Lochman, Stoloff, and 
Allbritton, 1972; and Katz, 1971); and (e) the cost of training fail- 
ures (Sands, 1971). These factors (which are not all listed, but in- 
dicative of the problem) are not considered in any one mathematical 
model reviewed. In fact, it is difficult to find a mathematical model 
that considers any two of them. 

Even if such factors could be adequately considered in a quantita- 
tive formula of some kind, it is extremely difficult to quantify all 
the costs associated with operating an OJT program. The Array, Navy, 
and Air Force have made some laudable efforts to do this (Arzigian, 
1967; Clary, 197C; Gay, 1973; Kollin, 1966; and Strope, 1967), but no 
military analyst has really been satisfied that he has accomplished 
what he set out to do. 

Cost-effectiveness studies of OJT are complicated by the fact that 
the OJT costs, per ae^ do not really provide enough information to 
make decisions. It is the relative costs of various options (e.g.. 
Plan A versus Plan B) that need to be determined before people can make 
the types of decisions that need to be made. 

One of the most important relative cost decisions is that about 
the mix of OJT and resident school training. Because the problem is 
such an important one (millions of dollars are involved), and because 
the information is urgently needed (military training loads have 
fluctuated widely in recent years) , the most sophisticated work on 
cost-effectiveness of OJT has been done in this area (Bateman, 1966; 
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Black and Bottenbt;rg, 1970; Gay, 1973; Lecznar, 1972; Weiher and 
Horowitz, 1971A, 1971B; Weinberg, 1967; Dunham, 1972; and Walker, 
1965). The results, however, are somewhat contradictory. One recent 
Navy study suggests that OJT is more expensive than resident school 
training because of all the supervisory time spent with OJT trainees 
(Weiher and Horowitz, 1971B), 

An equally recent Air Force study finds that the resident school 
training in Category B specialties is about twice as expensive as OJT, 
while the difference between graduates is negligible (Dunham, 1972) . 
Neither of these studies claims to have considered all the factors that 
should be considered in a cost-effectiveness study — a situation that is 
typical of work in this area— so the inconsistency can be explained in 
a variety of ways. 

One of the reasons that training cort-exfectiveness studies tend 
to be oversimplified is that the people performing the studies are not 
training experts. A more complete list of variables is provided by 
Walker (1965), who asked 20 training and human engineering personnel 
to develop their own criteria. The list of effectiveness variables 
that they mentioned is sizeable and much more numerous than the mathe- 
matically oriented cost-effectiveness studies reviewed in this report 
(Table V-1). 
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TABLE V-1 



Selection Criteria for Training 

Percentage of. Technical 



Trair^'ng Personnel IVho 
Selection Criteria Mentioned These Criteria 

1. Time to produce 100 

2. No* of students taught 83 

3. Effectiveness of teaching motor skills 75 

4. Effectiveness of teaching theory 75 

5. Amount of data needed to prepare 75 

6. Cost to teach 75 

7. Cost to develop 75 
8* Facilities needed to present 67 
9. Specific objectives 67 

10. Level of student intelligence 58 

11. Student knowledge 58 
12t Student Motivation 33 

13. Quality of specialists to develop material 33 

14. Ease of evaluating students 25 

15. Facilities needed to prepare 25 

16. Ease of administration 17 

17. Complexity of the training material 17 

18. Flexibility 17 

19. No. of instructors to inst.v^ct 17 

20. No* of instructors to develop 17 

21. Student Reinforcement 17 

22. Student Participation 17 

23. Realism^ 8 

24. Transfer of Training^ * 

25. Maneuverability 8 

26. Variety of psychological/ learning processes 8 

27. Student feedback 0 

28. Retention 0 

29. Student paced 0 

30. No. of senses stimulated 0 

31. Length of course ^ 0 

32. Stimulation of the operational environment 0 

33. Ease of presentation 0 

34. Competition 0 



From Walker, Ralph W. (1965). 

^Items in italics are management-centered items; others are 
student-centered items. 

^Equally student- and management-centered items, 
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S30) 

In the Air Force, basic technical skills classified as Category B 
skills are those in which a portion of the total personnel requirement 
is formally trained in a technical training course and the remainder 
is trained on the job (OJT). Determination of the relative numbers of 
airmen to be trained in formal courses and in on-the-job training is 
based, in part, i^on the time required to qualify a specialist at the 
five-skill, or fully qualified level. This rqport provides informa- 
tion on rates of progression to the five-skill level in Category B 
skills by comparing sanples of technical school and on-the-job train- 
ing personnel. Development of the methodology used to measure the 
rates of progression to the five-skill level was described, along with 
an explanation of the achievement ratio defined in this study. 
Achievement rates from basic military training (B^f^) graduation to 
award of the five-skill level were investigated. In the majority of 
the Category B specialties, there was inconclusive evidence of any 
advantage for technical training over OJT. In the specialties in 
which there were substantial differences between the two groups, the 
differences in most cases favored technical training. There were two 
specialties in which neither training group was superior to the other. 
The achievement rates after award of the three-skill level (i.e., ap- 
prentice level) to award of the five-skill level were not entirely 
consistent with achievement rates from B^f^ graduation to the five-skill 
level. In many cases, it was found that OJT personnel progressed more 
rapidly than technical school personnel from the three-skill to the 
five-skill level. ^-Auth. 
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Since 1965, when large numbers of American troops began to arrive in 
Vietnam, the draft has become a major public policy issue that has 
generated considerable literature. Most of these publications are; 
(1) analyses of the operation of the Selective Service System, (2) 
cost estimates of a proposed volunteer military force, or (3) polemics 
for a particular recruitment system. 

This study attempts to take an impartial and comprehensive look at the 
entire military recruitment issue. It asks: What are the attributes 
desired in a military recruitment system for the United States? It 
lists 17 criteria for assessing the relative merits of alternative 
recruitment sy stem. —Auth. 
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WOS 71-1, Naval Personnel Research and Development Laboratory, 
Personnel Systems Research Department, Washington Naval Yard, 
Washington, D.C., July 1970^ (AD 711 315) 

Reports FY 69 data on enlisted training time and costs to be used as 
inputs to the Career Premium Computer Program for determining Pro Pay 
eligibility. Specifically, it presents the training time and fiinds 
invested in enlisted personnel from initial procurement through ap- 
propriate basic training for 80 general and service ratings ajid ad- 
vanced and specialized training for 1,067 Navy Enlisted Classifica- 
tions (NECs) by source ratings. -->lut/z. 
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Level in the Conmmiaations Center Operations Specialty, 
Technical Report AFHRL-TR-72-56, Personnel Research Division, 
Air Force Human Resources Laboratory, Lackland AFB, Texas, 
June 1972. AD-753 093. 

Decisions concerned with the use of alternative Air Force training 
methods require several types of data. Among these are capacity to 
train, cost of the training, and quality of the trained airmen. The 
two methods of formal training in the Air Force are on-the-job train- 
ing (OJT) and technical school training. The data currently being 
provided to decision makers for selecting the proper mix of these 
two training methods can be substantially improved.— >lMt/i« 

Gay, Robert. "The Cost of On-rhe-Job Training in Military Occupa- 
tions," R-1351-ARPA, prepared for the Advanced Research Projects 
Agency under Project No. 9303, Rand, Santa Monica, Calif., 
April 1974. 

This Report is part of Rand's DOD Manpower and Training Management 
Program, sponsored by the Human Resources Research Office of the 
Defense Advanced Research Projects Agency (ARPA). The purpose of 
this research program is to bring new methodologies to bear on 
present and future military manpower problems. The purpose of this 
report is to provide a way of estimating the costs and determinants 
of on-the-job training in military occupations. As such, this study 
is only a pilot effort, meant to find a feasible technique for 
estimating these implicit--but none the less real-costs faced by 
the Department of Defense. The results from this pilot effort sug- 
gest that on-the-job training costs are more than twice as much as 
technical schooling costs for the occupation examined, aircraft 
maintenance in the Air Force. Thus, further study of these costs 
across occupations and services is warranted. In addition, this 
study is the first to estimate the relationship between individual 
characteristics and individual on-the-job training costs. Such in- 
formation, if validated by further study, may prove valuable to the 
military services with respect to selection, assignment, and pay 
policies. -->lut/i. 
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Hanushek, Eric A. Training in the Air Force— The Example of Graduate 
Education^ Research Report 71-3, U.S. Air Force Academy, 
Colorado, April 1971. 

This paper applies models for rational investing in human capital to 
Air Force decisions on training and particularly decisions about 
graduate education for officers. Two separate questions are addres- 
sed in depth. First, among the possible types of officers (rated- 
nonrated, reserve-regular, by length of service) who could be sent to 
school, are there any economic advantages to a specific group? In 
terms of the possibilities for the investment paying off (i.e., re- 
coi5>ing the costs of training) the decision rules are unambiguous: 
nonrated are a better investment than rated, and the more junior of-- 
ficer is a better investment than the more senior. Further, there 
is some reason to favor reserve officers rather than regular officers, 
since their retention is lower and lower retention in this case is 
desirable in terms of the total costs per year of educated service. 
Second, what is the trade-off between educating Air Force officers 
and hiring people who have purchased their own education? Here it 
spears isuch more expensive to educate Air Force officers; the long- 
run total cost differential between no internal education and all 
internal education is on the order of $100 million per year. While 
there are some compelling reasons to do some educating, serious con- 
sideration should be given to the question of how much educating the 
Air Force should sponsor .--i4wt/i. 

Katz, Aaron. Personnel Reactions to Incentives^ Naval Conditions 
and Experiences (PRINCE): Demographic and Background Informa- 
tion^ Expectations^ Attitudes^ Values and Motivations of Neu) 
Recruits^ WRR 72-2, Naval Personnel Research and Development 
Laboratory, Washington Navy Yard, Washington, D.C., September 
1971. ^ 

As part of a longitudinal study of the reactions of enlisted men 
to personnel incentives. Naval conditions and experiences, informa- 
tion on backgrounds, expectations, attitudes, values, and motivations 
were obtained at entry into the Navy from 6,795 recruits in Cate- 
gories I, II, and III on AFQT. 

New recruits see the Navy more frequently as an environment in 
which they can satisfy some short term goal. Some of the most im- 
portant reasons for joining the Navy were: to get technical train- 
ing, needing time to find out what they wanted to do with their 
lives, and not wanting to wait until they were drafted. 

On the basis of a comparison of: (1) their educational achievement 
to date with that of their parents, and (2) their occupational goals 
with the current occupations of their fathers, it appears that many 
recruits are interested in upward social mobility. -->lwt;i. 



Kim, Kwan H. , Lieser, Tom K. , Scheirer, William K., and Minckler, 
Rex. An Amy 75 Personnel Pvocurement Concept for the Depart-- 
ment of the Army, Volume I: The Study Report, Battelle Memorial 
Institute, Washington, D.C., 31 July 1968. 

Kollin, George. Phase I^-An Examination of Costs ard Recording 

Practices at CONARC Service Schools, Technical Paper RAC-TP-204, 
Research Analysis Corporation, McLean, Virginia, May 1966. 

The need for a study of training costs has often been expressed by 
Army personnel. Such a study would be of help to managers of plan- 
ning and programming of training especially when significant changes 
occur in the size and composition of force requirements and weapon 
systems. 

The two major areas of formal school training are the service schools 
and the training centers. Other large training areas are nonschool, 
such as on-the-job training and basic combat training. 

This study examines several service schools with respect to their 
training-cost recording practices, the degree of uniformity from 
school to school, and the cost elements that are considered. In ad- 
dition to determining what is occurring at the service schools, an 
attempt is made to indicate other cost factors related to training, 
but not now included by the schools. 

As a research strategy, courses for three hard-core military opera- 
tional specialties (MOSs) were chosen so that through the study of 
these individual courses the above objectives could be achieved more 
easily. 

To lend realism to the study, dollar costs for training these three 
skills have been included, but only to point up the complexities in- 
volved in a future attempt to arrive at complete, accurate training 
costs and a training-cost information system.— i4wifc^i. 

Lecznar, William B. The Road to Work: Technical School Training or 
Directed Duty Assignment? AFHRL-TR-72-29, Personnel Research 
Division, Air Force Human Resources Laboratory, Lackland AFB, 
Texas, April 1972. ArH754 845. 

This study explored the question of differences between ainnen who 
were assigned to jobs following graduation from fomal resident train 
ing schools and those who entered a field as on-the-job trainees. 
Eight career fields with a substantial number of airman input of low 
ability personnel under Project 100,000 were studied. Evaluations 
of cechnical school graduates and directed duty assignees were made 
in terms of six criteria- a job difficulty index, average task dif- 
ficulty, number of tasks performed, job interest, self- report of 
utilization of talent and training, and overall performance rating 
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with type of membership categories (i.e., resident technical course 
or direct assignment) and K^'th aptitude held constant, ^ests of the ' 
significance in difference becveen regression lines were made. In 
nearly every instance, the results indicated that the intercepts and 
slopes of the lines for the two groups were the same. That is, tech- 
nical school graduates and directed duty assignees were not different 
on any of the six criterion comparisons e This is not to suggest, 
however, that formal school training can be wholly displaced by on- 
the-job training. --i4ut/z« 

Lockman, Robert F., Stoloff, Peter H. , and Allbritton, A. Slagle. 
Motivational Factors in Aaaesaion and Retention Behavior, Re- 
search Contribution 201, Center for Naval Analyses, Institute 
of Naval Studies, Arlington, Va. , January 1972. 

Motivational factors in accession and retention behavior of Navy men 
were identified in a reanalysis of past surveys. Economic, psycho- 
logical, and personal history variables were found to be of joint 
importance in predicting enlistment and reenlistment behavior. Bet- 
ter measurement of these kinds of variables should result in im-- 
proved predictions and policy control mechanisms. --i4utfe« 

Mincer, Jacob. "On-the-Job Training: Costs, Returns, and Some Im- 
plications," Journal of Political Economy, Vol. 70, No. 5 (Pt. 
2, Suppl.), 1962, pp. 50-79. 

McCall, John and Wallace, Neil. Training and Retention of Air Force 
Aimen: An Economic Analysis, Memorandum RM-5384-PR, prepared 
for U.S. Air Force Project RAND, The Rand Corporation, Santa 
Monica, Calif., August 1967. 

This memorandum examines several aspects of the training and retention 
of Air Force electronic specialists. The major goal of the study is 
to investigate the responsiveness of the reenlistment rate to changes 
in Air Force remuneration. An attempt is also made to measure the 
degree to which Air Force training is transferable to the civilian 
economy. 

O'Flaherty, John and O'Rorke, Richard J., Jr. Costing a Unit Train-- 
ing Program: Volume I^^General Systems Description, Report 
RAC-R-23, Research Analysis Corporation, McLean, Va., April 
1967. (AD 814 958L) 

This report sets forth a methodology for ascertaining the direct 
variable costs incurred (parts and fuel consumption) through the 
usage of equipment in combat-unit training activities. Such costs 
are under current financial systems not readily available. Were 
such costs more readily available, through the adoption by the Army 
of a methodology such as described in the report, it is very possible 
that improvements could be effected in the usage of equipment, and 
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the broad objective to which such usage is directed--conibat-training 
readiness--could, in some degree, be enhanced. The report is pre- 
sented in two parts: Volume I- General Systems Description; and 
Volume II. Derivation of Parts Cost Factors. The first volume 
describes the purpose of the investigation, the problems encountered, 
and the methodology developed—the latter in general terras. Specific 
recommendations are presented on continuing the development of the 
system described in order to achieve an operational system within the 
Army. 

The second volume treats in detail the development of parts cost 
factors for 16 major items. The actual results are presented and 
analyzed and the computer programs are described. Problem areas are 
discussed and additional computer programs needed for improved ac- 
curacy and for theater, region, and unit factors (in addition to 
worldwide) are outlined, --i4Mt/z. 

Sands, William A. AppUaation of the Cost of Attaining Personnel 
Requirements (CAPER) Models WTB 72-1, Nnval Personnel Research 
and Development Laboratory, Wash^ '^g^^o" Navy Yard, Washington, 
D,C, , August 1971. 

The CAPER Model appears to be a proirising personnel system management 
tool. The output, numbers of men and dollar costs, is readily under- 
stood by everyone, unlike various statistical indexes popular among 
psychologists (e.g., validity coefficients). By requiring an 
explicit estimate of various types of costs, the model decreases the 
likelihood that policy decisions will be based upon implicit, un- 
recognized, and unwarranted assunqptions. Finally, the model enables 
the personnel manager to readily adapt his recruiting-selection 
strategy to changes in quotas, costs, and/or alterations in the re- 
cruiting environment. The flexibility that the CAPER Model provides 
to the personnel manager is enhanced by the computer program. --i4wt^. 

Strope, Donald H. "The Costs of Army Training, in Smith, R. Arthur 
and Allsbrook, Ogden 0, (edsO> Th^ Utilization of Military Re- 
sources-''' A Department of the Army Cost Research Seminar on 
Operating Costa, U.S. Government Printing Office, Washington, 
D.C., 1967, pp. 221-243, (AD 824 660) 

The purpose of this paper is to present some results of Research 
Analysis Corporation research on Army individual training costs in 
order to discuss implications for future research and for Army 
action necessary to improve estimates of Army training costs. As is 
generally known, there is currently inadequate knowledge to determine 
the resource implications of training to establish various personnel 
skills. 

It is important to know the context in which training costs will be 
used. The way costs are developed depends upon the type of problem 
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for which the costs are used. Different problems require different 
levels of accuracy and different degrees of input detail. Training 
costs are used in each of the types of problems shov/n in Chart 1, 
but the amount of training cost detail varies with the type of 
problem. -->lMt?z. 

Walker, Ralph W. ''An Evaluation of Training Methods and Their Char- 
acteristics," Human Factors , Vol. 7, No. 4, August 1965, pp. 
347-354. 

Sixteen training techniques were rated by experienced training per- 
sonnel each with respect to 34 training selection criteria. Training 
personnel tend to narrow their selection criteria to those elements 
that are administratively and contractually imposed. Educational/ 
psychological principles tend to be overlooked. Selection of train- 
ing techniques in practice should be on a broader and more systematic 
basis if the selection is to be properly justified. -->lut?z. 

Weiher, Rodney and Horowitz, Stanley A. A Production Function for 
Trained Recruits, CNA Professional Paper No. 84, Center for 
Naval Analyses, Arlington, Va. , November 1971. 

Weiher, Rodney and Horowitz, Stanley A. The Relative Coats of Formal 
and On-the-Job Training for Navy Enlisted Occupations , Profes- 
sional Paper No. 83, Center for Naval Analyses, Arlington, Va., 
November 1971. (AD 734 857) 

Weinberg, Gary. Army Training Cost Study (Draft report). Research 
Analysis Corporation, McLean, Va. , July 1967. 

The basic objective of the study was the development of a cost- 
estimating technique that can be used to estimate the cost of formal 
school training for military personnel. These estimates of formal 
school costs were generated for both individual MOSs and aggregated 
groups of manpower. The individual MOSs are those that involved 
formal school training in FY66. 

The nine Department of Defense occupational areas were used as the 
framework for the aggregated moJel. Within each MOS and occupational 
area, three skill levels were utilized-- junior, intermediate, and 
senior. These levels correspond to a defined mix of length of ser- 
vice, pay grade and MOS skill level. 

Cost totals were derived for each Army budget appropriation and they 
represent the total dollars generated by formal school training dur- 
ing FY66. An allocation scheme was developed to obtain a per-man 
course cost. In addition, turnover rates for both the MOSs and oc- 
cupational areas were formulated, enabling a cost analyst to de- 
termine a replacement training course cost. 
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A conparison was drawn between the MOS costs ^ occupational area 
costs and a per capita cost to show that, depending upon the level 
of aggregation used in any study, training costs may affect alter- 
natives under consideration. — Auth. 
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Recent years have seen an increasing emphasis upon the need for- 
statements of instructional objectives that are derived from job per- 
^'ormrt'^e and that are in a sufficiently operational language so as to 
genex; e test situations, whereby instruction can be evaluated (e.g., 
Wallis, 1966). However, it is one thing to advocate training based upon 
task analysis, but quite another to indicate a general method of 
analyzing tasks for this purpose --as the extent of the literature on 
task analysis testifies. If there is a weak link In the technology of . 
training it is surely here. — Auth. 

Benveniste, Guy. The Economice of the Hew Educational Media, Inter- 
national Institute for Educational Planning, Paris, February 
1965. (This is a working document, it is circulated for comment 
only.) 

When we speak of the "new media" we usually mean two kinds of 
teaching devices the mass media such as film, radio, and television 
used for instruction; and the new self-teaching devices such as pro- 
gra^saed instruction and language laboratories. 

This report explores the economics of instructional television, 
school radio broadcasting, films, and programmed instruction. Our 
objective is to provide a basis for determining the economic feasi- 
bility of using these new media in developing countries. 

We have emphasized the economics of those educational media that 
can be used to provide a substantial portion of a given curriculum. 
Tie report does not cover the economics of audio-visual aids in 
general . 

This research has recently been initiated at the Institute and is 
still underway. The purpose of this first or interim report is to 
bring up to date various individuals and institutions cooperating in 
this undertaking, and to >btain comments on the assumptions and 
methodology adopted and on the conclusions reached. 
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Bershtein, J. Design of an Enlisted Personnel Cost Analysis 

System, PRL Report No. WRM 67-11, Personnel Research Laboratory, 
Naval Personnel Support Activity, Washington, C,, October 1966. 
(AD 643 965) 

This memorandum provides in some detail the procedures and con- 
cepts of the enlisted personnel cost analysis system developed by the 
Personnel Research Laboratory. Sources of information and problem 
areas encountered during the course of research have also been in- 
dicated. — Auth* 

Carpenter, M. B. Maintaining Efficient Training Programs for Air Force 
TecJmical SpecialtieSj Report R-527-PR, The Rand Corporation, 
Santa Monica, Calif., September 1970. 

This Report sets forth suggestions for research on ways to 
better maintain the efficiency of Air Force training programs, based 
on k study of Air Force policy and practice in this area. The study 
addresses the problem from two points of view: What should be the 
steps in the initial selection of subject matter and in the design 
of a training program, given that the job to be trained for already 
exists in a fairly stable form? and. How can the subject matter be 
kept current? — Auth. 

Carpenter, Margaret B. and Haggart, Sue A. Cost-Effectiveness Analysis 
for Educational Planning, Paper P-4327, The Rand Corporation, Santa 
Monica, Calif., March 1970.' (AD 704 778) 

Champion, Joel T. and Ducharme, Richard E. An Evaluation of the 

AFM 66-1 Manhour Reporting System, SLSR-1-69, a thesis in partial 
fulfillment of the requirements for Master of Science degree. 
School of Systems and Logistics, Air Force Institute of Technol- 
ogy, Wright-Patterson AFB, Ohio, August 1969. (AD 863 839) 

AFM 66-1 Maintenance Management, provides a man-hour reporting 
system to aid maintenance managers in efficiently utilizing their man- 
power resources. However, this system became unpopular, and was made 
optional in 1965. An evaluation of its history indicated that it was 
a good system, but was misused and not fully understood by most 
maintenance managers. An analysis of labor force productivity since 
1965 showed: (1) that actual productivity has not reached manage- 
ment's expectations, and (2) that the use of the Man-Hour Reporting 
System could improve labor force use. The study recommended that 
this system be made mandatory and that the past problems identified 
he avoided. — Auth. 
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Clary, James N. Training Tim and Costs for Navy Ratings and NECSj 

WRM 68-13, Personnel Research Laboratory, Naval Personnel Program 
Support Activity, Washington, D. C, January 1968. (AD 667 578) 



This memorandum reports Fiscal Year 1967 data on enlisted training time 
and costs to be used as inputs to the Career Premium Computer Program 
for determining Pro Pay eligibility. Specifically, it presents the 
training; time and funds invested in enlisted personnel from initial 
procurement through appropriate basic Class ••A'^ training for 79 general 
and/or service ratings, and advanced and/or specialized training for 
707 Navy Enlisted Classification Codes .—Auth. 

Clary, James N. Training Time and Costs for Navy Ratings and NECs, 
WSS 69-'^, Naval Personnel Research and Development Laboratory, 
Washington, D. C, April 1969, 

Reports FY 68 data on enlisted training time and costs to be used as 
inputs to the Career Premium Computer Program for determining Pro Pay 
eligibility. Specifically, it presents the training time and funds 
invested in enlisted personnel from initial procurement through 
appropriate basic training for 80 general and/or service ratings and 
advanced ar.d/or specialized training for 888 Navy Enlisted Classifi- 
cations OiECs). '-Auth. 

Clary, James N. Training Time and Cost for Selected Ratings and NECs^ 
PRL Report No. WRM 67-20, Personnel Research Laboratory, Naval 
Personnel and Program Support Activity, Washington, D. C, 
December 1966. (AD 646 238) 

Reports FY 66 data on enlisted training time and costs to be used as in- 
puts to the Career Premium Computer Program for determining Pro Pay 
eligibility. Specifically, it presents the training time and funds in- 
vested in enlisted personnel from initial procurement through appropri- 
ate basic training for 28 ratings and advancement and/or specialized 
training for 171 Navy Enlisted Classification Codes .--Auth. 

Dupuy, H. J. and Deimel, R. W. Navy Recruitment Survey, WRR 68-1, 
Personnel Research Laboratory Naval Personnel Program Support 
Activity, Washington, D. C, September 1967. (AD 663 564) 

This report describes the results of a survey that was conducted to 
obtain information on the motivating influences affecting the decision 
to enlist or reenlist in the Navy. 

Survey questionnaires were mailed to U. S. Navy recruiting stations 
throughout the country with the instruction that the recruiter in 
charge administer them during the week of 17-22 July 1967 to the first 
five enlistees or reenlistees (no females) immediately following their 
•enlistment or reenlistment . Questions on this survey concerned spe- 
cific motivations for enlisting or reenlisting in the Navy. A space 
was also provided for a write-in response if the main reason (s) for 
one's enlisting were not mentioned in the survey. 



Findings reveal that ••Opportunity to get technical training/' "Desire 
to travel," "Desire to ftilfill military obligation at own time of 
choice," and "Desire for a Navy career" were the most important per- 
sonal reasons of the enlistees or reenlistees to enter the Navy. The 
influence of "Friends in service" and "Parents" were also important 
motivating forces. 

The results provide the most important reasons and motivations for 
entering the Navy at the present time, — Auth. 

Fortun^^, Jim C, Petry, John R,, and Harding, Larry G. Investigation 
of the NepA for Motion in the Teaching of a Complex Motor Skilly 
SRR 71-5, Naval Personnel and Training Research Laboratory, San 
Diego, Calif., August 1970, 

This report describes a comparison of a medium that allows motion 
(video-tape) and one that doesn't (slide-tape) in the teaching of 
complex motor skills (lockwiring) . Because of the differences in cost 
and in convenience of administration, it was deemed desirable to see 
if the video-tape produced significantly better results. 

The results showed no significant differences in achievement, length 
of time to perform the tasks, clarity of presentation, or pacing of 
instruction. It was concluded that complex motor skills can be 
adequately taught by a slide- tape presentation that requires the 
student to perform the motor skills as the proper procedures are 
presented. 

Garbutt, D. "The Costing and Evaluating of Training," in Barber, 
J. W., (ed.)> Industrial Training Handbook^ A. S. Barnes S 
Company, New York, 1969, p- 94-107. 

Gettings, R. Proposed Content of an Enlisted Pensunnel Cost Models 
PRL Report No. WRM 67-lR; Personnel Research Laboratory, Naval 
Personnel Program Support Activity, Washington, D. C, December 
1966. (AD 646 217) 

This report presents an outline of a proposed enlisted personnel cost 
model. The need to develop such a coniputerized cost model has become 
evident with the increasing number of requests for cost data from a 
wide variety of Navy and Department of Defense offices. 

The approach has involved an analysis of the present enlisted personnel 
cost system in terms of input, processing, and output. Revised outputs 
have been designed to provide concise reports of information frequently 
required. Modifications to the training cost reports are recommended 
as necessary to provide the revised outputs. — Auth. 

Goldfarb, Robert S. The Evaluation of Goverment Programs: Tlte Case 
of New Haven's Manpower Training Activities^ Ph.D. Dissertation, 
Yale University, New Haven, Conn., 1968, 260 pp. (PB 182 173) 
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This thesis attempts to deal with difficult methodological 
problems of progress evaluation; the particular programs considered 
are two types of manpower training programs, institutional or ''class- 
room^* training and government-subsidized on-the-job training, as they 
operated in New Haven from 1964 to 1966. 

Serious deficiencies of cost-benefit analysis are investigated, 
and the method is found to be inadequate for use in analyzing manpower 
programs. In particular, its tendency to ignore the existence of 
learning processes, which allows it to judge the potential of mature, 
experienced, efficient programs by studying new and inexperienced 
programs, is a critical weakness. 

Since estimating "capacity for learning*' or improvement is crucial 
for evaluating the potential of a program, methods of evaluation must 
be found which allow at least some of this capacity to be discovered. 
One way to do this is to study the way in which a particular process, 
such as a training program, turns inputs into outputs. Instead of 
ignoring the production process itself, as benefit-cost analysis tends 
to do, we must explore it. 

Studying the production process leads to a shift in the types of 
results the research produces. Rather than trying to discover whether 
classroom training or on-the-job training as it was run in New Haven 
was a "worthwhile*' investment (a question which we show that cost- 
benefit analysis actually fails to answer), we produce information 
that can contribute to the designing of better programs. Thus, the 
study itself becwnes a part of the learning and maturing process that 
produces better programs. 

Two data sources are used to study the production process. First, 
a group of trainees who underwent each type of training were inter- 
viewed. Second, employers who participated in the OJT program were 
interviewed, as were a group of the largest employers in the New Kaven 
area.— >lut/i. 

Glennan^ Thcisas K., Jr. Evaluating Federal Manpower Programs: Notes 
and Observations, RM-5743-0E0, prepared for Office of Economic 
Opportunity by The Rand Corporation, Santa Monica, Calif. , 
September 1969, 55 pp. (ED 041 111) 

Impact evaluations of manpower programs have had many short- 
comings, especially in finding control groups for comparison. Method- 
ological bias and inconsistency between evaluators, along with dis- 
interest by program administrators, have prevented evaluation from 
reaching its potential in program planning. The use of longitudinal 
studies to solve control group problems, with standardized criteria 
for benefit-cost analysis, could eliminate much of the inconsistency 
in evaluation, with improved information systems at the local project 
level, the evaluator and policymaker together could plan projects on 
the basis of accurate comparisons. —S.tf. 
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Grirasley, Douglas L, Acquisition^ Retention^ and Retraining: Effects 
of High and Low Fidelity in Training Devices^ Technical Report 
69-1, The George Washington University, Human Resources Research 
Office, HumRRO Division No. 3, Presidio of Monterey, Calif,, 
February 1969a, 

To examine the effects of varying fidelity of training devices on 
acquisition, retention, and reinstatement of a procedural task, 
soldiers were trained individually to operate the Section Control 
Indicator (SCI) console of the Nike Hercules guided missile system 
during preparation and firing status. Subjects with no previous 
experience on the equipment were trained on one of three panels 
differing in appearance, functional fidelity, or both, and tested 
immediately after training. Approximately four and six weeks later, 
they were retested and retrained to the original level of proficiency. 
Results indicated that there was no difference in training time, 
initial performance level, amount remembered after four and six weeks, 
or retraining time, between individuals trained on high- and low- 
fidelity devices for procedural tasks. — Anth. 

Grimsley, Douglas L. Acquisitiony Retention^ and Retraining: Train-- 

ing Category IV Personnel with Lew Fidelity Devices^ Technical 
Report 69-12, The George Washington University, Human Resources 
Research Office, HumRRO Division No. 3, Presidio of Monterey, 
Calif., June 1969^. 

Low (AFQT Mental Category IV) aptitude subjects with no previous expe- 
rience on the equipment were trained individually to operate a guided 
missile control panel. Three panels differing in appearance and/or 
functional fidelity were used. Subjects were tested immediately after 
training, four and six weeks later, and then retrained to the original 
level of proficiency. The results indicated that the higher aptitude 
subjects (from data presented in earlier STRANGER reports) required 
significantly less training time tham the low-aptitude subjects. For 
all treatment groups there were no practical differences in training 
time, initial performance level, amount remembered after four and six 
weeks, or retraining time between groups trained on high- and low- 
fidelity devices for this procedural task. Thus, training device 
selection should be based on a careful review of the tasks to be taught 
in order to use inexpensive devices where possible. — Auth. 

Hennessy, D. E, "Profit From Training,*' Training in Business and In- 
dustry , Vol. 6, No. 2, 1969, pp. 34-40. 

Henry, G. L. Determination of Manpower Cost Implications Associated 

With Changes in Navy Reenlistment RateSy WRM 71-30, Naval Personnel 
Research and Development Laboratory, Washington Navy Yard, 
Washington, D. C, March 1971. (AD 724 657) 
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This is the second, and final report of a study in which: (1) a 
practical means for calculating and displaying cost implications asso- 
ciated with changing Navy reenlistment rates was developed, and (2) the 
developed methodology was used to calculate reenlistment cost impli- 
cations associated with 1% changes in FY 72 first and second term 
reenlistment rates for each Navy enlisted rating* — Auth* 

Hook, Marion E. and Massar, Richard S» Rankorder Estimates of the Time 
Required for Crosstraining Among 98 Airmen SpeoialtieSj Technical 
Documentary Report PRL-TDR-62-15, 6570th Personnel Research 
Laboratory, Aerospace Medical Division, Lackland AFB, Texas, 
August 1962. (AD 290 551) 

When data based on experience are not available, it is often possible 
to obtain estimates of the relative time required to crosstrain per- 
sonnel qualified in one specialty to equal proficiency in a second 
specialty. This paper reports the first study on devising efficient 
methods for collecting and analyzing such estimates. A procedure was 
developed for collecting data to form a matrix describing the relative 
crosstraining-time demands for movements among a group of specialties. 

Rank order estimates of the crosstraining-time requirements for the 
9506 movements possible among the 98 5- level specialties were obtained 
from 477 Command and Staff College student officers. A computer 
program for hierarchical grouping was applied to these data to cluster 
specialties into groups such that crosstraining-time between special- 
ties within groups is minimized. Mean crosstraining-time estimates 
were computed at each stage of the clustering procedure as a criterion 
for evaluating the cost of reducing the number of clusters. For 
illustrative purposes, the hierarchical structure of the 40-cluster 
stage is compared with the 40 career-field designations of the 98 
specialties. — Auth. 

Hummel, Lester F. and Newraaster, Ronald D. Computerized Preparation 
of Navy Training Manuals^ Report 95, Operations Analysis Depart- 
ment, Navy Fleet Materials Support Office, Mechanicsburg, Pa., 
17 January 1973. (AD 754 431) 

Navy training manuals require continuing revision in order to remain 
current. Considerable time and man-effort are required to accomplish 
the revision of such manuals. The use of a computer assisted document 
preparation system in the updating of these manuals offers potential 
for significant reductions in both the elapsed time and the man-effort 
required to complete a revision cycle. This report relates the results 
of a pilot project initiated to test the feasibility of using such a 
computer-based text-editing system to update Navy training manuals. 
The Computerized Specification Management System (CSMS), a text-editing 
system developed by Naval Electronics Laboratory Center (NELC), San 
Diego, California, was used to prepare revised editions of two Navy 
training manuals and their associated correspondence courses. Modifi- 
cation and enhancement of the original system was required to provide 
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additional font and character capabilities. Significant reductions in 
elapsed time and man-effort required to update a manual were achieved. 
Additional texts will be assigned for update via the CSMS system and 
efforts to refine and improve the system will continue. — Auth. 

Hunter, Harold G., Lyons, J. Daniel, MacCaslin, Eugene F., Smith 

Robert G., Jr., and Wagner, Harold. The Process of Developing 
and Improiring Course Content for Military Technical Training^ 
Technical Report 69-9, The George Washington University, Human 
Resources Research Office, HumRRO Division No. 1, Alexandria, 
Va., May 1969. 

Curriculum development procedures in use as of 1966 for first- 
enlistment technical training in the Army, Navy, and Air Force are 
analyzed. A model process for training curriculum development was 
defined from training research findings and practices: (a) analyze 
the system, (b) develop task inventories, (c) develop a job model, 
(d) analyze its tasks, (e) derive training objectives, (f) develop 
the training program, and (g) monitor the trained product and modify 
the curriculum. A comparison between this model and the training 
development procedures in use in the service indicated a need for: 
(a) better procedures for determining the adequacy of training content 
and the means for improvement; (b) detailed guidance for developing or 
conducting the first four steps of the model process, criteria for 
allocating training content to formal instruction or on-the-job learn- 
ing, performance specifications for graduates, and feedback from 
training programs; and (c) more opportunities for career fields in 
training. — Auth. 

Kopstein, Felix F. and Seidel, Robert J. "Computer-Administered In- 
struction Versus Traditionally Administered Instruction: 
Econanics , in Atkinson, C. and Wilson, H. A. (eds.). Computer-' 
assisted Instruction: A Book of Readings^ Academic Press, 
New York, 1969, pp. 327-362. 

An attempt is made to assay the economics of computer- administered 
instruction (CAI) versus traditionally administered instruction (TAI) 
in controlling the structure of the learner's stimulus environment in 
teaching and training situations. There is a discussion of the need 
for a sound, objective economic appraisal of the value to society as a 
whole of increments in the breadth and depth of education in the popu- 
lation, and of the influence of varying rates with which these incre- 
ments are brought about. The necessity for reliable, objective 
information concerning cost data is emphasized. Projected cost/ 
effectiveness comparisons based on the assimiption of equal effective- 
ness for CAI and TAI are discussed for both civilian and military 
instruction . — Auth. 

Lefkowitz, J. ''Effect of Training on the Productivity and Tenure of 
Sewing Machine Operators," Journal of Applied Psychology ^ Vol. 
54, No. 1, 1970, pp. 81-86. 
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It was felt that a significant determinant of the high rate of per- 
sonnel turnover among a population of female sewing machine operators 
was inadequate initial training. Two hundred and eight new trainees 
received either 1, 2, or 3 days* vestibule trainings The longer the 
training, the lower the turnover rate, but the lower the productivity 
as well. Both effects were statistically significant, but the effects 
on productivity were deemed of less practical significance. A fourth 
training group received 3 days integrated vestibule and on-the-job 
training and achieved the best balance of productivity and employee 
retention. The more difficult the operation on which a trainee was 
placed the more likely she was to terminate her employ. Another sig- 
nificant determinant of resignations was employees' encountering a 
job that was contradictory to their expectations. — Autk. 

Naval Training Device Center, Proceedings of the Fourth Annual Naval 
Training Device Center and Industry Conference (Held 18-20 
November 1969), NAVTRADEVCEN IH-173, Naval Training Device 
Center, Orlando, Fla., 1969. 

A compilation of papers on a variety of technical subjects relating to 
training device technology. The conference theme, "Cost Effectiveness 
of Training Devices," provided a common ground for the exchange of new 
ideas and discussion of mutual problems. This fourth annual confer- 
ence is part of a continuing program tr encourage and develop better 
liaison between the Naval Traiixing Device Center and the trainer in- 
dustry. — Auth. 

Nowrasteh, Daryush M. Planning and Management Systems for State 

Programs of Vocational and Technical Education: An Application 
of Research, ERIC Clearinghouse on Vocational and Technical 
Education, Information Series No. 48 (VT 013 638), the Ohio State 
University, Columbus, Ohio, November 1971. 

O'Neill, Dave M. "Determinants of Labor Turnover Costs in the 

Military," Study 4, Part 1, Volume I of Studies Prepared for the 
President's Corrrdssion an an All^Volunteer Armed Force, U. S. 
Government Printing Office, Washington, D. C, November 1970. 

Pieper, W. J., Swezey, R. W. , and Valverde, H. H. Learner- Centered 
Instruction (LCI): Volume VII. Evaluation of the LCI Approach, 
Technical Report AFHRL-TR-70-1, Training Research Division, Air 
Force Human Resources Laboratory, Wright-Patterson AFB, Ohio, 
February 1970. AD-713 111. 

The evaluation of the Learner Centered Instruction (LCI) approach to 
training was conducted by comparing the LCI F-lllA Weapons Control 
Systems Mechanic/Technician course with the conventional Air Force 
course (ABR 32231R) for the same Air Force Specialty Code (AFSC) 
32231R on the following dimensions: (1) job performance of course 
graduates, (2) man-hour and oollar costs of the two courses, and 
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(3) student acceptability and instructor problems for the LCI course. 
Measures of job performance included a job performance test, an Air 
Force practical test, the supervisors' ratings, and a substitute job 
knowledge test. The graduates were measured both at en'^ of course 
(EOC) and again after five months in the field at field follow-up 
(FF). The high aptitude LCI trainees' job performance was superior 
to the high aptitude conventional course trainees at both EOC and FF, 
and job performance of the medium aptitude LCI trainees was about the 
same as that for the high aptitude conventional course trainees. Cost 
in terms of man-hours and dollars for the LCI course were substan- 
tially lower than those for the conventional course. The LCI course 
was about equally acceptable to the high and medium aptitude trainees, 
but some of the instructors had misgivings about the LCI approach. 
This report includes implications of integrating LCI courses into the 
Air Force training environment, — Auth. 

Reinhart, Bruce and Blomgren, Glen H, Cost^Benefit Analyaia—Tvade 
and Technical Education, Final Report, Division of Vocational 
Education, University of California at Los Angeles, and Bureau 
of Industrial Education, California State Department of Educa- 
tion, Sacramento, Calif,, August 1969, 82 pp, (ED 034 056) 

This report contains introductory material on cost-benefit analy- 
sis, reviews of cost benefit studies by Andrew J, Corazzini and by 
Jacob J, Kaufman and others, a discussion of the basic concepts of 
cost-benefit analysis, and a discussion of two possible approaches to 
cost-benefit analysis of vocational education: (1) vocational versus 
academic education, and (2) vocational versus vocational education, 
which includes on-the-job training costs for graduates of the various 
curriculums. The report concludes: (1) the typical study thus far 
has been limited in scope; an implicit problem has been that of com- 
paring different means as alternate means to the same ends, (2) the 
vocational versus vocational education approach would render unlike 
curriculums comparable, and (3) cost-benefit analysis is a useful de- 
cisional tool for allocating funds, but it assesses only the economic 
efficiency of a program, A plan for a possible cost-benefit analysis 
study of vocational education is presented in the appendix, --J. -K, 

Rhode, W, E,, Esseff, P, J,, Pusin, C, J,, Quirk, F, B, , and Shulik, R, 
Analysis and Approach to the Development of an Advanced Multi- 
media Instructional System^ Technical Report AFHRL-TR-69-30, 
Training Research Division, Air Force Human Resources Laboratory, 
Wright-Patterson AFB, Ohio, May 1970, 

In order to examine the possibilities for an advanced multimedia 
instructional system, this study begins with a comprehensive review and 
assessment of current instructional media in terms of: (1) a functional 
description, (2) instructional flexibility, (3) support requirements, 
and (4) costs. Following this, a model of an individual instructional 
system is developed as a basis for further analysis. Final comparisons 
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and **trade-offs" among the media are then made to arrive at a recom- 
mended media configuration that could serve as a multimedia base for 
an individualized instructional system. At this point, requirements 
and features of an automated management information and control sub- 
system to provide necessary operational control of the total instruc- 
tional system are outlined and discussed. Finally, the main features 
of a generalized plan for the development of such a system are de- 
scribed. --4ut/z. 

Sewell, David 0. Critique of Cost-Benefit Analysis of Training,'* 
Monthly Labor Review, Vol. 90, No. 9, September 1967, pp. 45-51. 

This edited version of a critique of cost-benefit analysis illu- 
mines the difficulties encountered in measuring the returns from on- 
the-job training accruing to the individual, society, and government. 
Its hypothesis, that findings from studies on the Manpower Development 
and Training Act (MDTA) projects do not transfer to the War on Poverty, 
supports a request for funding a cost-benefit analysis on training the 
poor under the Manpower Improvement Through Community Effort project 
in North Carolina. It criticizes generalizations that have been based 
upon three published analyses of training schemes conducted under the 
MDTA Act of 1962, the Area Redevelopment Act of 1961, and earlier 
State equivalents." It pointed up such difficulties as: (1) treating 
populations of the poor and unemployed as the same, (2) considering 
that improvement in income or employment are not the only benefits, 
and (3) trying to calculate the value of complementary demands for 
labor created by using the newly trained, --Auth. 

Short, J. G. and McCombs, J. L. A Study of the Feasibility of Using 
Fmgrarmed Instructional Techniques in U, S, Navy Correspondence 
Courses, Final Report AIR-D70-2/66-FR (Contract NOp-1514 with 
Bureau of Naval Personnel), American Institutes for Research, 
Pittsburgh, Pa., February 1966. (AD 803 984) 

The positive findings of the study indicate that there would be 
some increase in training effectiveness and efficiency if programs 
were used in Navy correspondence courses. Therefore, developmental 
and administrative costs of programmed and conventional courses were 
estimated, and comparative cost curves were prepared showing total 
training costs for different student populations. Program development 
costs are much higher than the present development costs of conven- 
tional courses. However, administrative costs associated with scoring 
assignments and a ^swering student questions are estimated to be lower 
for programs. Although these administrative costs are small for one 
student and one assignment, they represent a major cost factor because 
of the large number of students enrolled in Navy correspondence 
courses. There are certain student populations for which the higher 
initial developmental costs of programs can be partially offset by the 
reduction in continuing administrative costs . --Auth. 
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Smith, Robert G., Jr. The Engineering of Educational and Training 
Systems y D. C. Heath and Company, Lexington, Mass., 1971. 

Swann, J. H. Procedures for Determining Nmber of Instructors for Navy 
Enlisted Schools, PRL Report No. WRR 67-65, Personnel Research 
Laboratory, Naval Personnel Program Support Activity, Washington, 
D. C, June 1967. 

This report presents the final results of research concerned with the 
development of procedures for determining the number and indicating the 
4uality of instructors for Navy enlisted schools. The results are 
presented in the form of an instrument designed to be administered by 
schools to establish their instructor requirements. 

The instrument contains procedures for establishing the man-hour work- 
load of instructors as derived from their instructional, preparation, 
related, and military duties; and then calculation of the number of 
instructors required to perform the established man-hour workload. 

The instrument appears to provide a valid, practical, and operationally 
feasible basis for determining instructor requirements for Navy enlisted 
schools. The instrument has been recommended for promulgation. --i4ut/2. 

Swann, J. H. Procedures for Determining Number of Instructors for Navy 
Enlisted Schools y WRR 68-6, Personnel Research Laboratory, Naval 
Personnel Program Support Activity, Washington, D. C, November 

1967. (AD 824 246) 

Tilley, K. W. ''Studying the Cost-Effectiveness of Training," in Wilson, 
N. A. B. (ed.). Manpower Research (Proceedings of NATO Conference, 
London, August 14-18, 1967), English Universities Press Ltd., 
London, 1969, pp. 348-360. 

Tupes, E. C, Dieterly, D. L. , Fortuna, A. L. . and Madden, H. L. Devel- 
opment of a Data Base for an AFROTC Management Control System, 
Technical Report AFHRL-TR-68-118, Personnel Research Division, Air 
Force Human Resources Laboratory, Lackland AFB, Texas, December 

1968. AD-eSS 539. 

This report describes the origin and rationale of the concept of an 
AFROTC Management Control System, and the development of a data base 
upon which such a system must depend. A detailed list and descriptions 
of all variables in the data base are included. Some example distribu- 
tions are included to illustrate the type and magnitude of differences 
existing between the various AFROTC detachments. It is concluded that 
substantial improvements in the cost-effectiveness of the AFROTC pro- 
gram are possible through the use of the AFROTC Management Control 
System, but that the interrelationships between the various factors 
entering into such a system are so complex that the use of an elec- 
tronic computer in the data analyses is a necessity. --/lut/i. 

61 



Tupes, £• ^ . and Madden, H, L. The 1969 Updating of the Data File for 
the Aj?ROTC Management Control System, Technical Report AFHRL-TR- 
70-18, Personnel Research Division, Air Force Human Resources 
Laboratory, Lackland AFB, Texas, June 1970, AD- 709 729. 



This report describes the background and rationale of an AFROTC 
Management Control System and the expansion and updating through 
1969 of the data base upon which a system must depend. A detailed 
list and description of 40 detachment effectiveness criteria is pre- 
sented, with distribution data for 25 criterion variables. Effective- 
ness criterion scores for selected detachments are described, as are 
quality control tables developed to indicate the relative effective- 
ness of each detachment on each criterion. Also described is an 
effectiveness criterion prediction which makes available predicted 
scores for more than 800 colleges not presently in the AFROTC pro- 
gram. Examples are shown to demonstrate the usefulness of the 
system in the pre -evaluation of management decisions . --ylut/z. 



Warmbrod, J. Robert, Review and Synthesis of Research on the Economies 
of Vocational Education, Information Series, fhe Center for Voca- 
tional and Technical Education, ERIC Clearinghouse on Vocational 
and Technical Education, The Ohio State University, Columbus, 
Ohio, November 1968. (ED 023 937) 

Warren, M, W. ''Estimating Costs,'' Chapter 6, in Warren, M. W. , Train- 
ing for Results: A System Approach to the Development of Human 
Resources in Industry, Addison-Wesley , Reading, Mass,, 1969, 
pp, 89-105, 

Wethy, R, B. and Bumbak, A. Utilize Hon Cost of Capital Resources 
Used by Navy Training Schools: A Methodology, WRM 71-44, Naval 
Personnel Research and Development Laboratory, Washington Navy 
Yard, Washington, D. C, June 1971, (AD 727-102) 

This is a research study that developed a methodology for costing 
capital resources utilization (real property) by Navy training schools. 
The methodology was applied in several examples demonstrating the 
capability of the utilization cost methodology to provide Navy train- 
ing school managers at all levels with a more realistic and representa- 
tive total training cost. 

Wheaton, George R. and Mirabella, A. (American Institutes for Research). 
Effects of Task Index Variations on Training Effectiveness Cri-^ 
teria. Technical Report NAVTRADEVCEN 71-C-0059-1, Naval Training 
Device Center, Orlando, Florida, March 1972. 
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Wolff, H. H. "Cost Effectiveness of Training Devices," in Proceedings 
of the Fourth Annual NaOal Training Device Center and Industry 
Conference (held 18-20 November 1969), Naval Training Device 
Center, Orlando, Florida, 1969, pp. 1-4. 

Wood, W. D. and Campbell, H. P. Cost-Benefit Analysis and the 

Economics of Investment in Hwmn Resources. An Annotated BvbU- 
ography, Bibliography Series No. 5, Industrial Relations Centre, 
Queen's University, Kingston, Ontario, Canada, 1970, 217 pp. 
(ED 045 848) 



(For abstract see Section III.) 
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SECTION VI 



TECHNIQUE COMPARISON STUDIES 



Technique- comparison studies abound in the training literature. 
Among the moie interesting are those concerned with computer-assisted 
instruction (e.g., Kopstein and Seidel, 1969); learner-centered in- 
struction (e.g., Pieper, Swezy, and Valverde, 1970); the fidelity of 
training devices (e.g., Grimsley, 1969a, 1969b, 1969c); programmed 
instruction (e.g.. Short and McCombs, 1966); and multimedia techniques 
(e.g., Rhode, Esseff, Pusin, Quirk, and Shulik, 1969). General ad- 
vice on how to conduct such specific con5)arison studies has been pro- 
vided by Carpenter (1970), 

Unlike the mathematical models of OJT costs, the technique-com- 
parison studies are of great interest to trainers, and are reviewed 
on a selective basis in the abstract series published in the Joiamal 
of the American Society of Training and Development. It must be 
granted, however, that many people do not take this type of study 
very seriously. If the experimenter's conclusions disagree with the 
conclusions wanted by the training expert, the experimenter's con- 
clusions can usually be explained away with little effort. For one 
thing, the studies are almost always conducted by those who have a 
vested interest in an answer that supports the use of the new tech- 
nique. The experimental comparisons are almost never adequately con- 
trcll'^^d; and the number of cost-effective variables dealt with is 
invariably small. Many studies, for example, use "training time'' as 
an index of cost effectiveness, without measuring the relative per- 
formance of personnel in the experimental and control group. Many 
other studies evaluate trainee performance in a final course examina- 
tion, but provide no information about performance on the job. 

The number of variables that should be considered when two 
techniques a ^e compared is sizeable, A good illustration of what 
training experts are interested in is once again provided by Walker 
(1965) , who obtained expert ratings of 16 commonly used training 
techniques usinj, criteria proposed by training experts (Table VT-I). 
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Table VI-1 

Selection Criteria Matrix: Mean Evaluation Ratings^ 
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Grirasley, Douglas L. Acquisition^ Retention^ and Retraining: 

Effects of High and Low Fidelity in Training Devices^ Technical 
Report 69-1, The George Washington University, Human Resources 
Research Office, HumRRO No. 3, Presidio of Monterey, Calif., 
February 1969a. 

(For abstract, see Section V, Bibliography.) 

Grirasley, Douglas L. Acquisitions ^ Retention^ and Retraining: 

Group Studies on Using Lou Fidelity Training Devices. Technical 
Report 69-4, The George Washington University, Human Resources 
Research Office, HumRRO Division No. 3, Presidio of Monterey, 
Calif., March 1969b. 



Grimsley, Douglas L. Acquisition^ Retention^ and Retraining: 
Training Category IV Personnel With Low Fidelity Devices^ 
Technical Report 69-12, The George Washington University, 
Human Resources Research Office, HumRRO Division No. 3, Presidio 
of Monterey, Calif. , June 1969c. 

(For abstract, see Section V, Bibliography.) 

Kopstein, Felix F. and Seidel, Robert J. "Computer-Administered 
Instruction Versus Traditionally Administered Instruction: 
Economics," in Atkinson, Richard C. and Wilson, H.A. (eds.), 
Computer^Assisted Instruction: A Book of Readings^ Academic 
Press, New York, 1969, pp. 327-362. 

(For abstract, see Section V, Bibliography.) 

Pieper, William J., Swezey, Robert W. , and Valverde, Horace H. 

Learner-Centered Instruction (LCI): Volume VII. Evaluation 
of the LCI Approach^ Technical Report AFHRL-TR-70-1, Training 
Resear'"'- Division, Air Force Human Resources Laboratory, 
Wright-Patterson AFB, Ohio, February 1970. 

(For abstract, see Section V, Bibliography.) 

Rhode, William E., Esseff, Peter J., Pusin, Carol J., Quirk, Frank 

B. , and Shulik, Rubin. Analysis and Approach to the Development 
of an Advanced Multimedia Instructional System^ Technical Re- 
port AFHRL-TR-69-30, Volume I, Training Research Division, Air 
Force Human Resources Laboratory, Wright-Patterson AFB, Ohio, 
May 1970. AD-715 329. 

(For abstract, sec Section V, Bibliography.) 
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Short, Jerry G, and McCombs , J. Lynne. A Study of the Feasibility 
of Using Programmed Instructional Techniques in U.S. Navy 
Correspondence Courses^ (AIR-D70-2/66-FR) , Final Report for 
the Bureau of Naval Personnel (PTB 66-4) , American Institutes 
for Research, Pittsburgh, Pa. (AD 803 984) 

(For abstract, see Section V, Bibliography,) 

Walker, R.W. "An Evaluation of Training Methods and Their Charac- 
teristics," Human Factors ^ Vol. 7, No, 4, 1965, pp. 347-354. 

(For abstract, see Section V, References,) 

There are, of course, thousands of cechnique comparison studies 
in the training literature. Those that seemed to have special rele- 
vance for Air Force OJT programs are given here. 
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to Teaching Effectiveness of Technical Training Instructors^ 
Technical Report AFHRL-TR-70-8, Training Research Division, Air 
Force Human Resources Laboratory, Wright-Patterson AFB, Ohio, 
June 1970. AD-709 876. 

This study determined that field-experienced instructors teach- 
ing technical courses in Air Training Command produced no better 
students than did instructors lacking field experience. However, the 
field experienced instructors were rated higher by their supervisors 
and were considered more favorably by the students they taught. \TC 
has circulated this information to all of its training centers. — 
Auth. 

Biel, W.C. "Training Programs and Devices," in Gagng, R.M. (ed.). 
Psychological Principles in System Development j Holt, Rinehart 
§ Winston, New York, 1962, pp. 343-384. 
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Bond, Nicholas A., Jr. and Rigney, Joseph W. Measurement of Training 
Outcomes^ Technical Report No. 66, Behavioral Technology Labora- 
tories, University of Southern California, Los Angeles, Calif-, 
June 1970. (AD 711 302) 

Measurement of training outcomes is a requirement for evaluating 
new training techniques, but is one that is difficult to meet. 
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Managers of education and training may have different concepts of 
what they want, as favorable outcomes, than do the investigators 
doing the research. Classical statistical and experimental designs 
assume laboratory rigor of control over variables that is seldom 
possible in the real world of a school or classroom. Yet in the 
broader perspective of educational institutions, the effectiveness 
of these institutions is a current issue of fundamental concern in 
our society. In this report, possibilities for measuring outcomes 
of training are surveyed, considering training as a form of planned 
social change. Approaches that are disr^issed include the classic 
Solomon four-group design, iterative ad^tation to the peculiarities 
of individual student progress, response surface designs, adaptive 
control models, decision theory models, and simulation models. Illus- 
trations from the CAI literature of recent attempts to measure train- 
ing outcomes are given. The principal conclusions presented are that 
the classical four -way design is impracticable for most evaluation 
studies in training environments; that a policy of "adaptive research 
for big effects" is apt to be scientifically and administratively 
desirable; and that current attempts at measurement of training out- 
comes still use fairly simple methods. — Auth. 

Briggs, Leslie J. Sequencing of Instvuotion in Relation to Hievachies 
of Competence, Monograph No. 3, American Institutes for Research, 
Pittsburgh, Pa., 1968. 

Brock, John F. A Preliminaxy Investigation Into Shipboopd Training 
Problems. Final Report, SRR 72-1, Naval Personnel and Training 
Research Laboratory, San Diego, Calif., July 1971. (AD 726 689) 

A preliminary investigation into applying the NPTRL course de- 
sign procedure to a ship's repair party training program is reported. 
A training system was developed for repair party training which was 
workable and produced readily detectable improvements. The procedure 
proved adaptable to shipboard conditions. Most problems of shipboard 
training reduce to management problems, rather than the actual con- 
duct of training. There is tentative evidence that smaller repair 
parties, with personnel cross-trained in two or more discreet jobs, 
would be more efficient. Further research in the shipboard training 
area is suggested. -->lMt/i. 

Caro, Paul W. Equipment-Deviae Task Coimonality Analysis and Trans- 
fer of Training, Technical Report 70-7, Human Resources Re- 
search Organization, HumRRO Division No* 6, Fort Rucker, Ala., 
June 1970. 



Procedures were developed to enable training personnel to 
systematically and objectively determine the potential utility of 
training devices for teaching how to perform missions in operational 
equipment. The procedures allow comparison of operational task 



stimulus and response elements with corresponding elements in 
synthetic training equipment. On the basis of such information, 
training programs consistent with the psychological principles under- 
lying transfer of training may be developed. The procedures may be 
applied to the potential use of training equipment in a training 
situation other than that for which it was designed, or in determining 
the applicability of "off-the-shelP' training devices to specific 
training requirements. The procedures, termed Task Commonality 
Analysis, were developed in connection with an Army rotary wing in- 
strument flight training program. In an application of the procedures 
in that program, transfer of training predictions were generally con- 
sistent with empirical evidence collected earlier. — Auth. 

Childs, Gayle B. "Supervised correspondence instruction," in 

Wedemeyer, Charles A. (ed.). The Brandenburg Memorial Essays 
on Correspondence Instruction: I, University of Wisconsin, 
University Extension Division, Correspondence Instruction Pro- 
gram, Madison, Wise, 1963, pp, 22-33. 

Supervised correspondence study has made a major contribution 
to education in this country. It is equally clear that the full 
potential of the procedure is far from being real i zed. 

Cunningham, J.W. (ed.). The Joh-Cluster Concept and Its Curricular 
Implications: A Symposiim^ Center Monograph No. 4, Department 
of Psychology, North Carolina State University at Raleigh, N.C. , 
1969. 

Among the many problems facing occupational education today, one 
of the most pressing is that which relates to the analysis and clas- 
sification of jobs. In attempting to improve the quality of voca- 
tional education, it is necessary to know the characteristics of the 
product that is needed, as well as the process by which it is pro- 
duced. Research into the area of job-clustering has great potential 
for helping us to understand the nature of the desired end-product of 
vocational education. The potential for curriculum development is 
apparent, but that is not the only possible contribution. A better 
understanding of job-analysis, a necessary foundation for clustering 
research, should give greater feedback capabilities in occupational 
programs with a concomitant improvement in the ability to modify and 
redirect occupational programs to provide the best articulation with 
the world of work. The papers collected in this monograph represent 
the contributions of a number of scholars who are presently active in 
research on the problems of job-clustering. ---Aut/i. 

Davis, R.H. and Behan, R.A. "Evaluating System Performance in 

Simulated Environments," in Gagnd, R.M. (ed.). Psychological 
Principles in System Development, Holt, Rinehart § Winston, 
New York, 1962, pp. 477-515. 



Friedman, Herbert L. and Johnson, Raymond L. (American Institutes 

for Research). Time-^Compressed Speech as an Educational Medium: 
Studies of Stimulus Charaotevistios and Individual Differences j 
Final Report prepared for the U.S. Department of Health, Educa- 
tion, and Welfare, Office of Education, September 1969. 

Glaser, R. and Klaus, D.J. "Proficiency Measurement: Assessing 

Human Performance," Gagn€, R.M. (ed.). Psychological PHnciples 
in System Development, Holt, Rinehart § Winston, New York, 1962, 
pp. 419-474. 

Hooprich, E.A. and Matlock, E.W. PHnted-^Circuit-^Board Soldering 

Training for Group IV Personnel, SRR 71-11, Naval Personnel and 
Training Research Laboratory, San Diego, Calif., October 1970. 
(AD 713 639) 

As part of a larger program to determine which Navy skills can 
be learned by lower aptitude personnel and to ascertain what methodr 
are most effective for accomplishing such training, an experimental 
course in printed-circuit-board soldering was administered to 186 
Mental Group IV students in 13 classes. Two different training ap- 
proaches—one stressing instructor guidance and the other featuring 
reliance on film viewers— were evaluated. Research data were obtained 
by means of questionnaires and paper-and-pencil and performance 
tests. The soldering proficiency of the Group IVs was compared with 
that of experienced Navy technicians, and tests designed to measure 
transfer of the skills learned in the course also were constructed 
and administered to selected classes. The major research findings 
were: (1) that Group IVs can learn the rather exacting skills re- 
quired for a task such as printed-circuit-board soldering, but they 
require a considerably longer training period than other Navy per- 
sonnel; and (2) that the fiimviewer method of training is an effec- 
tive alternative to conventional methods of teaching soldering and 
was preferred by the Grotq) IV personnel. Recommendations for im- 
plementing the research findings into Navy technical training are 
presented, and other current and planned research projects are 
briefly discussed. ---Awt/z. 

Horowitz, Morris A. and Hermstadt, Irwin L. (Department of Econom- 
ics, Northeastern University). A Study of the Training of Tool 
and Die Makers, prepared for Manpower Administration, U.S. De- 
partment of Labor, Washington, D.C., September 1969. (PB 187-558) 

Hughes, John L. (ed.). Prograrmed Learning: A Critical Evaluation 

(A publication of the Foundation for Research on Human Behavior) , 
Educational Methods, Inc., Chicago, 1963, 238 pp. 
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Hummel, Lester F. and Newmaster, Ronald D. Corrputerized Preparation 
of Navy Training Manuals^ Report 95, Operations Analysis Depart- 
n»nt. Navy Fleet Material Support Office, Mechanicsburg, Pa., 
17 January 1973, (AD 754 431) 

Navy training manuals require continuing revision in order to 
remain current. Considerable time and man-effort are required to ac- 
conplish the revision of such manuals. The use of a computer- assisted 
document preparation system in the updating of these manuals offers 
potential for significant reductions in both the elapsed time and the 
man-effort required to conqplete a revision cycle. This report relates 
the results of a pilot project initiated to test the feasibility '^f 
using such a Computerized Specification Management System (CSMS)» This 
text editing system developed by Naval Electronics Laboratory Center 
(NELC), San Diego, California, was used to prepare revised editions 
of two Navy training manuals and their associated correspondence 
courses. Modification and enhancement of the original system was re- 
quired to provide additional font and character capabilities. Sig- 
nificant reductions in elapsed time and man-effort required to update 
a manual were achieved. Additional texts will be assigned for update 
by the CSMS system and efforts to refine and improve the system will 
continue. — Auth. 

Jackson, Rex. Developing Critevion-ReferenGed Tests ^ ERIC Clearing- 
house on Tests, Measurement, and Evaluation, Princeton, N.J., 
June 1970, 18 pp. (ED 041 052) 

Klaus, David J. and Glaser, Robert. Increasing Team Proficiency 
Through Training: 8. Final Simnary Report^ AIR-E1-6/68-FR, 
Team Training Laboratory, American Institutes for Research, 
Pittsburgh, Pa., May 1968. 

This report summarizes the results of research at the Team Train- 
ing Laboratory from December 1960 until August 1967. During this 
time, seven technical reports were issued by the laboratory. This 
summary report briefly describes each of these seven studies and re- 
views their purpose and major results. 

Report 1 described the approach being examined in the Team 
Training Laboratory, one which considered the team and its output 
or product rather than the performance of its individual members as 
the focus of investigation. 

Report 2 reported on the acquisition and extinction of a team 
response, a demonstration that basic principles of individual learn- 
ing could be applied to the team considered as a single entity. 
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Report 3 presented an experiment on the inclusion of parallel 
or ••redundant'' members in a team which confirmed an hypothesis de- 
rived from the underlying approach that redundancy could result in 
eventual decrements in team performance. 

Report 4 further analyzed the effects of internal team structure 
on the development and maintenance of a team response based upon the 
degree of correspondence between individual performance and feedback 
supplied to the team. 

Report 5 identified the relationships among team member char- 
acteristics, the conditions of team training and the speed and 
thoroughness with which teams developed proficiency that could be 
demonstrated empirically. 

Report 6 explained the value of more gradually introducing the 
low ratios of reinforcement typical of early team performance pro- 
viding supplemental, supervisory- furnished feedback to team members. 

Report 7 presented three studies on the simulation of team en- 
vironment, which considered the degree to which the approach facili- 
tated the replication of team learning phenomenon based upon the per- 
formance of a single individual. 

The final section of this report identifies some practical im- 
plications of this research and relates the underlying concepts to 
the broader context of social behavior. — Auth. 

Lee, Seong Soo. 'Transfer From Lower-Level to Higher-Level Concept," 

Journal of Verbal Learning and Verbal Behavior^ Vol. 7, 1968, 
pp. 930-937. 

Two experiments were carried out: one to test a hypothesis about 
the operation of the learned hierachy of concepts and the other to 
examine the function of attribute-coding responses. The main vari- 
able in Exp. 1 was the number of lower-level concepts to be learned. 
The main variable in Exp. 2 was the amount of pretraining on attri- 
bute-coding responses. 

Pretraining on all three of the lower-level concepts facilitated 
the acquisition of a presumed higher- level , biconditional concept 
significantly more than other conditions. Pretraining on the attribute 
coding responses prior to rule-learning proper also facilitated the 
acquisition of the transfer concept significantly more than did no 
training in coding. Thus, the second experiment yielded empirical 
support for an interpretation of the transfer effects apparent in 
the learned hierarchy of concepts in terms of two processes: (a) im- 
plicit or explicit attribute-coding, and (b) rule formulation 
proper. --i4u<:fe. 
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Main, Ray The Effectiveness of Flash Cards in a Mathematics Self-- 
Stiidy Course for Group IV Personnel, SRM 70-20, Naval Personnel 
and Training Research Laboratory, San Diego, Calif,, June 1970, 
(AD 707 718) 

The Naval Personnel and Training Research Laboratory is con- 
ducting research aimed at identifying optimal methods for training 
Navy personnel who have achieved marginal scores on military selec- 
tion tests. In the present study, flash card instruction methods 
were adapted for application to a comprehensive range of basic mathe- 
matical operations involved in a previously developed course in 
fundamental mathematics. Supplementing the standard course work with 
flash-card instruction did not result in significantly higher per- 
formance gains. It was concluded that applying flash methods to the 
relatively wide range of content complexity involved in this sti^dy 
was not effective. It is pointed out that this investigation should 
not be interpreted as challenging the usefulness of flash cards as 
typically employed. —Auth. 

Meyer, John K. and Abraras, Macy L. Development of a Strategy to 
Retrieve Information From the Standards Used to Evaluate Welds 
and Metals by Nondestructive Testing, SRR 71-20, Naval Personnel 
and Training Research Laborator>-, San Diego, Calif., April 1971. 
(AD 883 413) 

This study reflects the initial research effort to improve the 
training technology in the nondestructive testing (NDT) training 
course, A strategy was developed to retrieve information from the 
Welding Standard (NAVSHIPS 250-1500-1), one of the primary military/ 
Navy standards (documents) that is used to evaluate welds and metals 
by NDT methods. This strategy, an index and retrieval exercise, 
was chen experimentally evaluated by comparing groups using it with 
those groups not using it. Improvements were demonstrated in the 
capability of finding inspection topics, the interpretation of in- 
spection topics, and in the length of time required. Recommendations 
are made for the implementation of the index and the retrieval exer- 
cises, and for developing similar index documents and retrieval 
exercises for all standards involved in the NDT of welds and metals. — 
Auth. 

Nelson, Hilding E. National Institute on Innovative Curriculum in 
Vocational-Technical Edux^ation, Final Report, Vocational-Indus- 
trial Education Research Report, prepared by the Department of 
Vocational Education, Pennsylvania State University for Bureau 
of Research, Office of Education and Welfare, Washington, D.C., 
August 1969, 295 pp. 

This document reports on two institutes designed to communicate 
new concepts and procedures in vocational-technical education curricu- 
lum development to potential change agents in the various states. 
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Formal presentations by 12 consultants and task force activities con- 
cerned innovation identification and prognoses for change, planning 
strategies for curriculum innovation, climates for innovation and 
change, inq)lementing and expanding innovation, and cost-benefits and 
evaluation criteria. Institute evaluation is discussed in detail. 
Institute participants represented 31 states. Complete texts of 
formal presentations are appended — C.H. 

St, Michel, K,A. and Swanson, C.L, A Microfilm System for Improving 
the Dissemination of Mvy Occupational Information to Recruits: 
A Feasibility Study ^ SRR 69-18, Naval Personnel and Training 
Research Laboratory, San Diego, Calif,, March 1969, (AD 687 430) 

This research evaluated the feasibility of a microfilm system 
for providing information ^bout Navy ratings to recruits and evaluated 
the adequacy of Navy job information presently being given to recruits 
Additionally, it provided information about: (a) what job information 
is considered most important, (b) the effectiveness of different ways 
of learning about Navy occupations, and (c) the recruits' expressed 
satisfaction regarding job assignments. The microfilm system that 
was investigated was considered too costly for implementation in cur- 
rent recruit training. A recommendation for tryout and evaluation of 
a simplified microfilm method was made, --i4wt/z. 

Standlee, L,S, , Matlock, E,W. , and Harrigan, R.J, Development of 
Methods and Materials for Soldering Training^ SRR 71-19, Naval 
Personnel and Training Research Laboratory, San Diego, Calif,, 
February 1971, (AD 720 308) 

Two parallel self-instructional packages were developed for 
teaching soldering rework on printed circuit boards. One package 
consisted of 12 five-minute sound films. The other consisted of a 
168-page spiral bound book. The effectiveness of the self-instruc- 
tional packages was compared to that of a parallel instructor-taught 
course. Subjects for the comparison experiment consisted of 60 stu- 
dents in Basic Electricity and Electronics, 

Both self-instructional packages- - fi Im and book--were found to 
be effective media for learning soldering skills. No significant 
difference was found in the soldering skill gains of film-, book-, 
and instructor-taught students. The soldering book, though, has the 
advantage of being simpler and less expensive to reproduce and to 
administer, --i4ut/i, 

Taylor, John E, , Montague, Ernest K, , and Michaels, Eugene R, An 
Occupational Clustering System and Curriculum Implications for 
the Comprehensive Career Education Models HumRRO Technical 
Report 72-1, Human Resources Research Organization, HumRRO 
Division No, 3, Presidio of Monterey, Calif,, January 1972, 
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Design of a proposed occupational clustering system for the 
Comprehensive Career Education Model (CCEM) was to meet three general 
criteria: encompass most existing jobs, translate into an entire 
K-12 curriculum, and show clear advantages over other systems. Re- 
searchers examined existing clustering systems for relevance and pos- 
sible adaptation; no one system met all the criteria; so, a new clus- 
tering system was devised by synthesizing useful features of existing 
systems. The proposed system has two crucial dimensions— one stres- 
sing functions and contents of occupations, the other emphasizing 
status or levels of occupations. The proposed clustering system was 
planned to fulfill three instructional functions: inform students 
about the world of work, assist students in choosing a suitable 
career, and provide models to shape instructional objectives and 
learning experiences. --i4Mt^. 

Wedemeyer, Charles A. (ed.). The Brandenburg Memorial Eaaaya on 
Correspondence Instruation--! ^ University Extension Division, 
Correspondence Instruction Program, Madison, Wis.; 1963, 77 pp. 

In August 1961, the Brandenburg Foundation made its first mem- 
orial grant to The University of Wisconsin Extension Division for 
the purpose of assisting the university in carrying on seminars in 
correspondence instruction and publishing every other year a volume 
of original papers on correspondence instruction, of which this is 
the first. There is a very brief review of the literature of the 
field in the foreword. Titles, authors, and author locations of the 
collected papers are: "Liberal Education for Adults by Mail," by 
Leonard S. Stein, St. Louis University; "The Demands of the Decade," 
by John L. Davies, State University of Iowa; "Supervised Correspon- 
dence Instruction," by Gayle B. Childs, University of Nebraska; 
"Tutoring Through the Mail," by Katherine W. McMullen, University of 
Wisconsin; "Problems in Learning by Correspondence," by Charles A. 
Wedemeyer, University of Wisconsin; "W.H. Lighty— Fountain of Ideal- 
ism," by Roger W. Axford, University of Wisconsin; and "Some Aspects 
of Teaching by Correspondence in Australia," by Renee Erdos, New 
South Wales Department of Technical Education, Australia. --i4ut^. 

Wedemeyer, Charles A. (ed.). The Brandenburg Memorial Eaaaya on 
Correspondence Instruction—I I ^ University of Wisconsin Press, 
Madison, Wis., 1966, 156 pp. 

Various aspects of correspondence education are explored in 13 
essays by various authors, including the editor. In respective 
essays, current worldwide trends in correspondence education are 
summarized, its development in several countries is discussed, its 
purposes are explored, its weaknesses and advantages are analyzed, 
methods of evaluating it are proposed, and predictions about its 
future are offered. The contents are: "World Trends in Correspon- 
e^nce Education," by Charles A. Wedemeyer; "Correspondence Education 
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in Developing Countries/* by Homer Kempfer; "Teaching Through Tele- 
vision," by Harold Wiltshire and Fred Bayliss; "The Expansion of 
Educational Opportunity in Venezuela," by Pedro Tomas Vasquez; "New 
Developments in the Production of Swedish Correspondence Courses," 
by Borje Holmberg; "New Horizons in Correspondence Education," by 
Philip Lambert, Eldo C. Koenig and William 0* Vebber; "The Search for 
Purpose in Correspondence Education," by Ripley S. Sims; "Words with- 
out Gestures," by George Hartung; "The Role of the Instructor in 
Correspondence Study," by Margaret I. Knowles; "Promoting Correspon- 
dence Instruction," by Clarence A. Schoenfeld; "A Self-Evaluation 
Study of the Correspondence Method," by Harold Glen Clark; "Review 
of Research in Correspondence Study," by Gayle B. Childs; "Extension 
Education and Its Tools in the Next Half Century," by Charles A, 
Wedemeyer. — Auth. 

Wedemeyer, Charles A. "Problems in learning by correspondence," in 
Wedemeyer, Charles A., (ed.) , The Brandenburg Memorial Essays 
on Correspondence Instruction: I. University of Wisconsin, 
University Extension Division, Correspondence Instruction Pro- 
gram, Madison, Wis., 1963, pp, 46-54. 

Certain problems of the correspondence learner might possibly 
be special hazards of the correspondence method. These problems do 
not result from deficiencies in the method (for the problems appear 
in other methods, too), but ignoring these problems or failing to 
work imaginatively to overcome them may offer greater hazards for 
the correspondence student than for students of other methods.— i4wt;2. 
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SECTION VII 



SYSTEMS ANALYSIS OF TRAINING 



The training program design literature is very useful and inter- 
esting to training experts. This literature is especially important 
because of recent technical breakthroughs in the design of training 
programs that would tend to integrate program evaluation and txainmg 
program design as part of a "systems analysis" or "systems engineer- 
ing" approach to training. 

Systems analysis of training refers to a set of procedures for 
designing or redesigning programs of instruction. One might think of 
it as a feedback-oriented approach to the design of instructional pro- 
grams since, in essence, it consists of an analysis of the tasks the 
student is expected to perform following training; identification of 
the terminal behaviors that jointly make up the tasks; measurement of 
these terminal behaviors ; and revision of the program of instruction 
until performance specifications are met. In other words, the objec- 
tives of the program are defined at the outset, and the training pro- 
gram is revised repeatedly until it meets those objectives. A "sys- 
tems analysis" emphasis shows up in several ways. Job analyses are 
conducted from a systems point of view, and systems diagrams are used 
extensively in making decisions about training-program design. 

The Air Force version of systems analysis of training is called 
Instructional Systea Development (ISD) and is presented in Figure 
VI I -1. The Air Force approach has proved to be very effective, as 
indicated by the many complimentary comments in a recent OSD-sponsored 
analysis of curriculum development policies and procedures used by 
Army, Navy, and Air Force (Hunter, et al.).^ Moreover, the Air Force 
has improved its approach since the survey was conducted. 

The origins of systems analysis of training are difficult to 
trace, since so many different organizations seemed to have been work- 
ing on t^e problem at the same time. The Army and the Air Force were 
key organizations in developing the operational plans for systems 
engineering of training. The military experts who did the work based 



■^Hunter, Harold G. , Lyons, J. Daniel, MacCaslin, Eugene F., Smith, 
Robert G. , Jr., and Wagner, Harold. The Prooess of Developing and 
Improving Course Content for Military Technical Training, The George 
Washington University, Human Resources Research Office, HumRRO Divi- 
sion No. 1, Alexandria, Va. , May 1969. 
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it upon training research conducted by large behavioral science 
organizations, such as the Hxman Resources Research Organization 
(HumRRO) and the American Institutes for Research (AIR)--but the act 
of putting all the pieces together at one time into one systematic 
approach seems to have been conducted by committees and boards of 
military training experts who prepared systems engineering manuals 
for Army and Air Force use. The Navy has recently developed a similar 
manual for use by its schools and training centers. 

Systems analysis of training is expensive, and it has been ap- 
plied almost exclusively to the design of lengthy resident school 
training programs. Similar procedures are talked about in the design 
of OJT programs, but the responsibility for the design of these pro- 
grams is basically supervisory. One can tell a busy supervisor that 
he is supposed to conduct task analyses and think in terras of terminal 
and enabling performance objectives--but , generally, the supervisor 
siii5)ly does not have the time or the skill required to do a professional 
job of this sort. So as a result, the application of systems -oriented 
techniques to OJT is more of a long-range objective than it is a cur- 
rent reality. 

Many systems analysis manuals have been developed. The following 
Bibliography presents documents that qualify as important systems eng- 
ineering training manuals from diverse sources. 

BIBLIOGRAPHY 



Briggs, L.J. Handbook of Procedures for the Design of Instruotioriy 
American Institutes for Research, Pittsburgh, Pa., 1970. 

Butler, F.C., Jr. Job Corps Instruotional Systems Development Manual^ 
Rocky Mountain Educational Laboratory, Inc., Denver, Colo., 
1967. 

DeCecco, J.O. The Psychology of Learning and Instruction: Educa- 
tional Psychology^ Prentice-Hall, Inc., Englewood Cliffs, New 
Jersey, 1968. 

Deterline, W.A. Principles and Practice of Instruotional Technology: 
Workbook^ General Programmed Teaching, Palo Altc, Calif., 1968. 

Drumheller, S.J. Handbook of Curriculum Design for Individualized 
Instruction: A Systems Approo'ih^ Educational Technology Pub- 
lications, Englewood Cliffs, New Jersey, 1971. 

Gilbert, T.F. *'Mathetics: The Technology of Education, Journal of 
Mathetics, Vol. 1, 1962, pp. 7-73. 
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Cropper, G^L. and Short, J.G. Design of a Training Development System^ 
American Institutes for Research, Pittsburgh, Pa,, 1969. 

Hamrens, D.G. "Instructional Systems Development,'^ in Crawford, J. 
(ed.), CORD National Research Training Mamial^ Teaching Research 
Division of the Oregon State System of Higher Education, Monmouth, 
Oregon, 1969. 

Postlethwait, S.N. , Novak, J., and Murray, H.T. , Jr. The Audio- 
Tutorial Approach to Learning^ Burgess Publishing Company, 
Minneapolis, Minn., 1964. 

Rundquist, Edward A. Joh Training Coiirse Design and Improvement 

(2nd ed.). Research Report SRR 71-4, Naval Personnel and Training 
Research Laboratory, San Disgo, Calif., September 1970. 
(AD 876 204) 

The second edition of the course design manual is a thorough 
revision of earlier editions. The manual is designed to assist in- 
structors in developing and improving job-related training courses. 
Major changes from earlier editions include more careful definitions 
of training and training-related terms; a general clarification of 
concepts and procedures, especially those concerned with job and skill 
analysis for training purposes; more emphasis on principles of de- 
veloping training exercises; a more thorough consideration of the 
in5>ortance and means of adapting individual differences; and more 
emphasis on the significance of the course mission for course design. 
Examples from a wide variety of duty assignments are included.— i4wt/i. 

Smith, Robert G., Jr. The Design of Instructional Systems^ HumRRO 
Technical Report 66-18, The George Washington University, Human 
Resources Research Office, Alexandria, Va. , November 1966. 

Tosti, D.T. and Ball, J.R. A Behavioral Approach to Instructional 
Design and Media Selection^ Westinghouse Learning Corporation, 
Albuquerque, New Mexico, 1969. 

Tuckman, B.W. A Study of Curriculums for Occupational Preparation 
and Education (Scope Program: Phase 1)^ Rutgers University, New 
Brunswick, New Jersey, 1970. 

Twelker, P. A. ''Designing Instructional Systems,'' in Crawford, J. 

(ed.), CORD National Research Training Manual, Teaching Research 
Division of the Oregon State System of Higher Education, Monmouth, 
Oregon, 1969. 

U.S. Department of the Air Force. Instructional System Development , 
AFM 50-2, Headquarters U.S. Air Force, Washington, D.C., 1970. 
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U.S. Department of the Array, The Development of InatrucHonal Systems. 
Procedures Manual, U.S. Army Security Agency Training Center and 
School, Fort Devens, Mass*, 1967. 

U.S. Department of the Army. Systems Engineering of Training (Course 
Design) y U.S. Continental Army Command, Fort Monroe, Va. , 1968. 

U.S. Department of the Navy. Fundamentals of Navy Curriaulum Flan" 
ningy NAVPERS 93510-1, Bureau of Naval Personnel, Washington, 
D.C. , October 1968. 




SECTION Vlil 
APPROACHES TO PROGRAM EVALUATION 



The Air Force has been concerned with ways of evaluating train- 
ing programs for some time, and recently sponsored a literature review 
on this subject. The report (Bergman and Siegel, 197?) reviews the 
literature on training evaluation and student achievement measurement 
(316 references). This literature review was then used as the basis 
for the design of a ••how-to-do- it** manual that was made available to 
training experts as well as to researchers in the Air Force (Siegel, 
Bergman, Federman, and Sellman, 1972). These two documents, however, 
are oriented primarily toward resident school courses rather than 
OJT. 

The most important point to be made here is that there are many 
different approaches to the evaluation of OJT programs; Siegel et al. 
(1972), Rafacz and Foley (1973), and other experimental -oriented ex- 
perts stress the use of performance checklists and rating scales. 
Programs can also be evaluated in terms of their cost-effectiveness 
from an approach of planning programming and budgeting systems (PPBS) 
(Nowrasteh, 1971; Hennessy, 1969); the statistical characteristics 
of the overall program (lacobelli, 1971); the actual behavior of 
the program graduates (Hahn, 1970, 1973); and so on. 

The decision as to which program evaluation technique should be 
used under what circumstances is not an easy one. Rose (1968) main- 
tains that eight different portions of the training cycle should be 
evaluated with one or more of 13 different techniques. The eight 
elements requiring evaluation are job-task analysis, training re- 
quirements, plan for training, administration of program, formal 
training process, OJT process, end-of-course outcomes, and perform- 
ance on the job. The 13 recommended methods for evaluating these 
various training program elements are cited here. This list is a 
useful one, but probably contains no surprises for those in the train- 
ing business. 

(1) Comparison with the findings of educational 
research and the psychology of the learning process. 

(2) Comparison with performance requirements on the 
job. 

(3) Observation of instruction, 

(4) Written achievement tests. 

(5) Questionnaire to trainees. 
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(6) 



Questionnaire to supervisors. 



(7) 


Trainee interview. 


(8) 


Supei^isor interview. 


(9) 


Observation of performance on the job. 


(10) 


Study of records and reports. 


(11) 


Performance tests. 


(12) 


Analysis of problems and accidents. 


(13) 


Research with matched groups. 



One of the problems with the typical approach to training evalu- 
ation is that it does not give adequate consideration to the view- 
point of the trainees. The objectives of the trainees differ from 
the objectives of the organization for which they work, and any 
evaluation process based solely upon organizational objectives is 
not complete. 

It is not, of course, always clear what the trainee objectives 
are. One study, a factor analysis of variables of interest to young 
trainees, produced the following factors: 

(1) Community adjustment. 

(2) Striving for personal improvement. 

(3) Realistic aspirations. 

(4) Job motivation and planning. 

These factors seem to have obvious implications for ''motiva- 
tional" problems of trainees — but they tend to be neglected when 
training programs are evaluated. 

Putting more emphasis on ratings by trainees is one way in which 
attention can be focused upon trainee objectives as well as organi- 
zational objectives. This approach is especially useful when gradu- 
ates of vocational training programs (e.g., U.S. Department of Labor, 
1970) evaluate the training that they have received since the grad- 
uates are in a better position to evaluate their training than they 
were as trainees. Graduates are also much less concerned about ad- 
verse reactions from their trainers when they make critical comments. 
A bibliography of program evaluation references, other than those 
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cited in the text, that seemed useful to those concerned with Air 
Force OJT programs is also given. 

REFERENCES 

Bergman, Brian A. and Siegel, Arthur I- Training Evaluation and 

Student Achievement Measurement: A Review of the Literature, 
Technical Report AFHRL-TR-72-3, Technical Training Division, 
Air Force Human Resources Laboratory, Lowry, AFB, Colo,, 
January 1972. AD-747 040. 

Hahn, Clifford P. Development of Task Level Job Performance Criteria. 
Interim Report ok PhnBe I, AIR-24300-2/73-IR, prepared for Air 
Force Human Resource^ Laboratory (Contract F41609-71-C-001 0; 
Item OOOIAA-CDRL Sequence No. A002) , by American Institutes 
for Research, Silver Spring, Md., 1973, 

This report describes the activities undertaken in the accomplish- 
ment of Phase I of a planned three-phase project designed to create 
and evaluate a methodology for the development of task-level job- 
performance criteria. The primary objectives of the project were the 
initial development and the later evaluation through field trials of 
instruments and procedures that could provide the armed services with 
an acceptable measure of job-task performance. The resulting criter- 
ion instruments and procedures were to be designed for use in per- 
sonnel management, systems development, planning, and evaluation 
rather than as a basis for immediate actions that affect the per- 
sonal careers or compensation of individual servicemen. Project 
activities were to be addressed to a limited number of Air Force 
specialty career ladders. The basic intent of the project, however, 
was the development of illustrative methodology that could be readily 
translated across many Air Force specialties, as well as to related 
occupational specialties across all military services. 

The scope of Phase I, which is covered in this report, included 
the selection of the occupational specialties to be studied, and the 
development and preliminary evaluation of sui*vey instruments and pro- 
cedures preparatory to a field administration to be accomplished in 
Phase II. --Auth. 

Hahn, Clifford P. "Methods for Evaluating Counter -Measure Interven- 
tion Programs," in Evaluative Research Strategies and Methods 
(proceedings of a seminar held 8-9 January 1970), American 
Institutes for Research, Pittsburgh, Pa., 1970. 

Hennessy, D.E. "Profit From Training," Training in Business and 
Industry y Vol, 6, No. 2, 1969, pp. 34-40. 
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lacobelli, J.L. Training in Private Industry: Policies^ Attitudes^ 
and Practices of Employers in Greater Cleveland^ Manpower 
Research Monograph No. 22, U. S. Department of Labor, Manpower 
Administration, 1971 . 

Nowrasteh, Daryush M. Planning and Management Systems for State 

Programs of Vocational and Technical Education: An Application 
of Research^ ERIC Clearinghouse on Vocational and Technical 
Education, Information Series No. 48 (VT 013 638), The Ohio 
State University, Columbus, Ohio, November 1971. 

Rafacz, Bernard A. and Foley, Paul P. Preliminary Results on the 
Evaluation of a Fleet Post-Training Performance Evaluation 
Technique^ WTR 73*10, Naval Personnel Research and Development 
Laboratory, Washington Navy Yard, Washington, D,C,, January 
1973. (For abstract, see p. 92, Bibliography.) 

Rose, Homer C. *'A Plan for Training Evaluation,*' Training and 
Development Journal^ Vol. 22, No. 5, 1968, pp. 38-51 • 

Siegel, Arthur I., Bergman, Brian A., Federman, Philip, and Sellman, 
Wayne S. Soma Tenhniquee for the Ei^aluation of Tanhnical 
Training Courses and Students^ AFHRL-TR-72-15, Technical 
Training Division, Air Force Human Resources Laboratory, Lowry 
AFB, Colo., February 1972. AD- 753 094. 

U.S. Department of Labor, Manpower Administration. Toward the Ideal 
Journeyman, Volume 1. An Optimum Training System in Apprentice- 
able Occupatiors, Manpower Research Monograph No. 20, Manpower 
Administration, Washington, D.C., 1970. 

BIBLIOGRAPHY 

American Institutes for Research Evaluative Research, Strategies 
and Methods, American Institutes for Research; Pittsburgh, 
Pa., 1970. 

In planning this seminar, it was originally intended as a meet- 
ing of AIR staff only; those interested and experienced in evaluation. 
Because AIR's work in the field, both in. this country and overseas 
is increasing, we sought to have staff within and between offices 
meet to discuss the timely topic of evaluative research. 

The AIR Board of Directors felt there were other groups and 
individuals who would be interested in the topic and who could pro- 
vide additional slants, and insights. It was agreed to invite a 
selected group of participants. The response was most rewarding; 
more tlian 200 accepted the invitation to meet on 8 and 9 January 1970 
in Washington, D.C., to discuss Evaluative Re^^^^rch; Strategies and 
Methods. 
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The objectives of the seminar were to provide opportunities 
for both staff and colleagues to gain 1) new insight or perspective 
on the broad problems of evaluation, and 2) additional knowledge 
about methods for dealing with these problems. 

This report includes the eight papers presented during the two- 
day meeting together with supplementary comments. The success of 
the seminar rested largely on the high quality of these presenta- 
tions. Dr. Arthur A. Lumsdaine, Chairman, Department of Psychology, 
University of Washington, commented on the first four papers. On 
the second day, the commentator was Dr. Irving Spergel, Chairman, 
Community Work Sequence, School of Social Service Administration, 
University of Chicago. The speakers and participants appreciated 
the added perspective these commentators provided* — Auth. 

Baldwin, Thomas S. tUnive.-sity of Illinois). The Development of 
Achievement Measures fox* Trade and Techniaal Education^ U.S. 
Department of Health, Education, and Welfare, Office of Educa- 
tion, Washington, D.C., Sept^ber 1970. 

Belasco, James A. and Trice, Harrison M. The Aasesmefit of Change 
in Tiaining and Therapy j McGraw-Hill Book Co., New York, 1969^ 
165 pp. 

Brooks, Carl N. 'Training System Evaluation Using Mathematical 
Models, Educational Technology, Vol. IX, No. 6, June 1969, 
pp. 54-61. 

This paper applies mathematical modeling techniques to the pre- 
design evaluation of the performance of an automotive mechanics 
training system. First, the method of deriving the model will be 
demonstrated in three stages: 

1. derive the flowchart model of the system as it should 
operate, 

2. convert flowchart functions into mathematical form; 

3. derive the equations describing the system performance. 

Second, the mathematical model will be applied in predicting the 
s/stetn performance. This application demonstrates the economic ad- 
vantages of using the mathematical modeling approach. — Auth. 

Brydle, John Robert (Wayne State University). "Analysis of Major 
Electronics Technician Training Problems Encountered by Leading 
Electronics Systems Manufacturers in the United States,** Research 
in Industrial Education, Summary of Studies 1960^-1961, Trade and 
Industrial Education Series No. 75, Vocational Division Bulletin 
No. 299, U.S. Department of Health, Education, and Welfare, 
Washington, D.C. , 1962. 
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The purpose of the study was to collect and analyze the major 
technical training problems of commercial organizations offering 
electronics technician training programs. 

Information was collected from 50 leading electronics systems 
manufacturers in the United States. 

The ten leading problems revealed by the respondents were: 

(1) The students often lack a good foundation in mathe- 
matics and basic sciences. 

(2) There was not enough time to train instructors in 
both theoretical and practical aspects of the equip- 
ment they taught. 

(3) Technical manuals were inadequate for training. 

(4) It was difficult to obtain enough laboratory equip- 
ment. 

(5) There was a shortage of instructors. 

(6) There was not enough time to train instructors in 
teaching techniques. 

(7) Better tests of the students' ability to maintain 
equipment were needed. 

(8) Training supervisors were not close enough to train- 
ing problems, and could not keep up to date on new 
equipment being taught. 

(9) Some students took their work too lightly. 

(10) Some instructors needed more field experience on the 
equipment they taught. — Auth. 

Caldwell, Lynton K. "Measuring and Evaluating Personnel Training," 
PuDlia Personnel Review, Vol. 25, No. 2, April 1964, pp. 97-102. 

"The ultimate test of training is whether it attains its objec- 
tives. But it is also important to know whether those objectives 
and the efforts to achieve them are worthwhile in terms of cost and 
in relation to the goals and objectives of the government. Because 
training entails costs and because it substantially affects the per- 
formance of the public service, any serious effort to improve the 
public service through training will make provision for measuring 
and evaluating training effectiveness... This discussion [by the 
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author] dispels the belief that the impact of training is too com- 
plex to measure and that the variables involved defy evaluation/' 
Contents: Establish performance standards; gauge for achievement 
needed; pitfalls in measuring results; selecting measurement cri- 
teria; keep tab on program's value; criteria defining values; avoid 
noting unstated objectives; and conclusions, — Auth. 

Coster, John K. and Ihnen, Loren A, ^'Program Evaluation," Chapter 
VIII in Review of Educational Research^ XXXVIII, No, 4,, 
October 1968, pp. 417-433, 

Craig, Robert L. and Bittel, Lester R, Training and Development 
Handbooky McGraw-Hill Book Co,, New York, 1967, 650 pp, 

Foley, John P., Jr, Performance Testing: Testing for What Is Realy 
AMRL Memorandum P-42, 6570th Aerospace Medical Research Labor- 
atories, Aerospace Medical Division, Wright -Patterson AFB, 
Ohio, June 1963, (AD 412 776) 

It is generally conceded that performance examinations are 
superior to written examinations for the measurement of job behav- 
iors required of Air Force specialists and technicians. However, 
performance examinations, both in school and on the job, are time 
consuming in their development and in their administration. Written 
examinations have, therefore, been substituted in many cases for 
performance examinations. This memorandum presents the difficulties 
involved in developing and administering performance examinations, 
the dangers of depending upon written examinations as substitutes 
for performance examinations, aad the fact that there is a lack of 
research information on the valid substitution of written for per- 
formance examinations. It proposes developmental research with a 
view of simplifying performance examination procedures and estab- 
lishing valid guidelines as to the scope of written examinations 
in measuring job behaviors. It, also> makes some recommendations 
concerning immediate action that can be taken to improve the validity 
of current technical training examination procedures, — Auth. 

Freeberg, Norman E, and Reilly, Richard R, Development of Guidance 
Measures for Youth- Work Training Program Enrollees. Phase I: 
Measurement of ProgitKon Objectives and the Development of 
Criteria^ PR-71-13, prepared for U,S. Department of Labor, Man- 
power Administration, by Educational Testing Service, Princeton, 
N.J., July 1971, 152 pp. 

Rationally defined outcome variables, for use in evaluating 
youth-work training programs, were incorporated in questionnaires 
administered to present and former Neighborhood Youth Corps enrol lees 
and analyzed to determine their suitability as criterion measures. 
Using a factor analytic technique, empirically defined clusters 
were obtained for immediately available (end-of-program) outcomes and 
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longer term (post -program) outcomes.. The most logical groups of 
criteria were found for those former enrol lees who had full-time 
employment experience. Relatively clear patterns of job-oriented 
capability and success were dominant as were two separate factors 
bearing on personal adjustment to the job and to the community. 

Some descriptive highlights concerning the vocational behaviors 
ot former trainees are presented and future research needs for better 
definition and understanding of program objectives are dis- 
cussed.— ^lut^z. 

Glaser, Robert. "Instructional Technology and the Measurement of 

Learning Outcomes: Some Questions," in Popham, W. James fed.), 
CrtteirLon-Referenoed Measurement, An Introduction, Educational 
Technology Publications, Englewood Cliffs, N.J., 1971, pp. 5-14. 
(See also same title, American Psychologist, Vol. 18. 1963 
pp. 519-521.) 

Glaser, R. and Klaus, D.J. "Prof iciency Measurement : Assessing 
■ Human Performance," in Gagne, R.M. (ed.). Psychological Prin- 
ciples vn System Development, Holt, Rinehart and Winston, New 
York, 1962, pp. 419-474. 

Glaser, Robert and Nitko, A.J. "Measurement in Learning and In- 
struction," in Thorndikc, R.L. (ed.). Educational Measurement, 
American Council on Education, Washington, D.C., 1971. 

Gustafson, Herbert W. Research on Methods of Evaluating Maintenance 
Proficiency, Technical Report AFPTRC-TR-58-6, Air Force Per- 
sonnel and Training Research Center, January 1958. (AD 152-108) 

This Technical Report summarizes briefly the findings of explor- 
atory studies, conducted under Task No. 37303, designed to better the 
measurement of the performance of maintenance personnel by improving 
testing procedures and use of information derived from records of 
job performance. Four separate research efforts were devoted to 
developing improved performance-testing techniques: (1) preparation 
of a guidebook on maintenance-performance evaluation; (2) statistical 
re-analyses of proficiency test data to relate troubleshooting per- 
formance to specific kinds of aptitudes and basic knowledge, and to 
determine the relations among types of errors committed in performing 
complex alignments and aljustments; (3) development of two micro- 
film projection devices for use in training troubleshooters and in 
assessing troubleshooting performance; and (4) a study of the effects 
of variations m performance -testing procedures. Two studies of 
maintenance performance records are also reported. 
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Hawkridge, David G, , Campeau, Peggy L,, Weisgerber, Robert A,, 

Manes^ Audrey L,, Fors, Robert A,, and Youngquist, Louise V. 
A Study of Selected Programs for Vooatioyuzl Education in 
Secondary Schools ^ Final Report AIR-848-1/70-FR, prepared 
for Office of Education^ U.S- Department of Health, Education, 
and Welfare by the American Institutes for Research, Palo ^Ito, 
Calif., January 1970, 95 pp, (ED 041 155) 

The aims were to identify, select, analyze, and describe voca- 
tional programs at the secondary level tha'w had been successful in 
increasing the total placement rate (employment, further education) 
when compared with other courses of instruction. Through a litera- 
ture search, mail and telephone inquiries, personal contacts, and 
other means, 445 programs were identified for study with onphasis 
on program evaluation, particularly follow-up of graduates. Some 
30 programs were site-visited. 

None of the 445 programs could be shown to have met the study's 
criteria for success; 43% lacked comprehensive follow-up data on 
graduates; 33% had nearly complete data, but no trends could be de- 
tected, nor were suitable comparison groups of nonvocational gradu- 
ates available. 

Even if comparable, contemporaneous groups had existed, and even 
if complete follow-up of both groups of graduates had been possible, 
it was concluded that the original criteria needed to be supplemented 
by criteria that took into account qualitative placement factors, 
such as initial job earnings, on-the-job competency, job retention, 
progress within the job, and mobility within a range of jobs related 
to the training provided by the vocational program. 

Recommendations for program design and evaluation that would 
permit the assessment of such factors conclude the report, — Auth. 

Jacobs, Paul I., Maier, Milton H., and Stolurow, Lawrence M, 

A Guide to Evaluating Self-Instructional Programs^ Holt, 
Rinehart and Winston, 1966, 

Kayloe, Alvin G. "A Method for Evaluating the Effectiveness of 

Technical Training," Training and Development Journal^ Vol- 25, 
No. 6, June 1971, pp. 24-30. 

Lee, Walter S. "A New Model for Psychological Services in Educational 
Systems," Educational Technology^ Vol. XII^ No. 6^ June 1972, 
pp. 22-24. 
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Little, J. Kenneth. Review and Synthesis of Research on the Placement 
and Follow-Up of Vocational Education Students^ Information 
Series, ERIC Clearinghouse on Vocational and Technical Education, 
The Center for Vocational and Technical Education, The Ohio 
Stace University, Columbus, Ohio, February 1970. (ED 037 543) 

Mager, Robert F. and Pipe, Peter. Analyzing Performance Problems 
or 'You Really Oughta Wanna\ Lear Siegler, Inc., Education 
Division, Fearon Publishers, Belmont, Calif., 1970. 

Mobilization for Youth, Inc. The Work Sample: Reality-Based Assess- 
ment of Vocational Potential ^ Final Report DLMA-82-34-69-21-4, 
prepared for U.S. Department of Labor, Manpower Administration, 
by the Experimental Manpower Laboratory at Mobilization for 
Youth, Inc., New York, March 1971. (PB 199 474) 

The purpose of the handbooks is to serve as a technical guide 
in establishing work sample programs. The publication is a "how-to" 
manual for developing appropriate work samples, establishing the 
vocational evaluation unit, and training vocational evaluators. The 
objective of the handbook is to aid intended users in assessing the 
vocational potential of the disadvantaged, — Auth. 

Moss, Jerome, Jr. and Stromsdorfer, Ernest W. "Evaluating Vocational 
and Technical Education Programs," in Somers, Gerald G. and 
Little, Kenneth (eds.), Vocational Education: Today and To-- 
rwrroWy Center for Studies in Vocational and Technical Education, 
The University of Wisconsin, Madison, Wise, 1971, pp. 219-261. 

Norton, Robert E., Love, Lamar E., and Rolloff, John A. Guide to 

Improving Vocational Education Evaluation^ College of Education, 
University of Arkansas, Fayetteville, Arkansas, Decanber 1970. 

Short-term training institutes represent one of a variety of 
methods being used to improve the status of vocational education 
evaluation. In addition to developing leadership competencies needed 
in evaluation, another purpose of these institutes has been to draw 
upon the talents of the consultants and participants in order to help 
develop improved strategies and procedures of evaluation. 

In keeping with these purposes, a major part of this guide is 
concerned with presenting viewpoints on which there was general con- 
sensus among the participants attending the National Institute on 
Improving Vocational Education Evaluation, which was held August 4-8, 
1969, at the University of Arkansas. Attention is also given to 
points and issues on which there seemed to be little agreement among 
either the consultants or the participants. Finally, some suggestions 
on how evaluation can \ , improved are offered. — Auth^ 
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Popham, W. James. ''Objectives and Instruction/' in Pophara, 

Eisner, E.W. , Sullivan, H.J., and Tyler, L.L. Instructional 
Objectives, American Educational Research Association Mono- 
graph on Curriculum Evaluation No. 3, Rand McNally, Chicago, 
1969, pp. 32-64. 

Rafacz, Bernard A. and Foley, Paul P. Preliminary Results on tke 
Evaluation of a Fleet Poct-Training Performance Evaluation 
Technique^ lirrR 73-10, Naval Personnel Research and Development 
Laboratory, Washington Navv Yard, Washington, D.C., January 
1973. 

The purpose of this research effort is to validate the utility 
and effectiveness of a unique human performance measurement technique 
developed under ONR contract (N0001467C0107) . Performance data on 
eight Navy ratings were collected from sh.ips of LANTFLT and PACFLT. 
This report is the first in a series of technical reports on the 
statistical analysis of that data. In particular, a statistical 
analysis is provided on performance-related data for electronic main- 
tenance personnel sampled from 11 ships of CRUDESFLOT NINE, San Diego, 
California. Four different performance estimators, as functions of 
critical incidents, are evaluated with respect to a performance cri- 
teria. A detailed explanation of the distributional properties of 
the performance estimators is presented and an explanation of the 
factors that lead to the adoption of a curvilinear regression an- 
alysis for analysis of the data is discussed. 

The results of the statistical analysis indicated that a cer- 
tain combination of the performance data possessed moderate validity 
for measuring the absolute level of technician performance. Detailed 
analysis of the performance data also identified the areas of diffi- 
culty that have to be avoided in order to improve upon the validity 
of each of the performance estimators. These and other preliminary 
results indicate that the technique possesses merit for further de- 
velopment and research. The Naval Personnel Research and Develop- 
ment Laboratory is continuing research on this technique with re- 
spect to the totality of data collected.— ilwt/z. 

Ratliff, Forrest R., Chiorini, John R., Curran, Charles R., and 

Shore, C. Wayne. Evaluating Combat Crew Training Performance 
Using Criteria of Minimum Performance Standards, AFHRL-TR- 
70-50, Personnel Division, Air Force Human Resources Labora- 
tory, Lackland AFB, Texas, November 1970. AI)~722 409. 

An 11-point rating scale and minimum acceptable performance 
criteria were developed for each training phase of F-4 combat crew 
training to measure training progress and compare the performance of 
two groups of student aircraft commanders. Instructor pilots rated 
the performance of student aircraft commanders who were either 
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upgrading second-seat crew members or recent undergraduate pilot train 
ing (UPT) graduates. The objectives were to determine the effects on 
performance in combat crew training school of experiences as F-4 sec- 
ond-seat crew members, and to determine whether recent UPT graduates 
with no experience as second-seat crew members could meet the minimum 
performance standards of combat crew training in the amount of time 
currently allotted. The students were rated against the criteria of 
miminum acceptable performance using standardized rating procedures. 
Several conclusions were reached on the basis of this study: (a) Pro- 
ficiency ratings made against criteria of minimum acceptable per- 
formance can be used to measure training progress and compare rates 
of gain in proficiency between two or more groups; (b) experience as 
F-4 second-seat crew members resulted in higher proficiency ratings 
at the beginning of most training phases for the upgrading second- 
seat crew members; (c) recent UPT graduates improved their rated 
performance to achieve levels of proficiency similar to the upgrad- 
ing second-seat crew members by the end of most training phases; 
and (d) students from the top 10% of their UPT classes should be 
able to successfully complete F-4 combat crew training as aircraft 
commanders in the amount of time currently alloted without previous 
exposure to the F-4 as second-seat crew members. — Auth. 

Rose, Homer C. The Development and Supervision of Training Programs^ 
American Technical Society, 1964. 

Schultz, Douglas G. and Siegel, Arthur I. Post-Training Performance 
Crite-rion Development and Application: A Selected Review t f 
Methods for Measuring Individual Differences in On-The-Job 
Performance, Applied Psychological Services, Wayne, Pa., July 
1961. 

For several years. Applied Psychological Services has been 
carrying out research in the development and application of criteria 
for assessing the proficiency of Naval technicians in various techni- 
cal specialties. Before undertaking additional work, it seemed wise 
to evaluate the current state-of-the-art with respect to methods for 
the measurement of individual differences in on-the-job performance. 
This report considered recent progress in the area and attempted to 
point up a number of important issues that require investigation and 
clarification. 

Job performance appraisal techniques that have been used were 
discussed. These included production records, interviews and ques- 
tionnaires, work sample and situation tests, appraisal of executive 
performance, and rating scales. Criterion analysis was reviewed 
in terms of intercorrelation and factor analysis, scaling, and re- 
liability, including job performance changes over time. 
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Important current issues in the field of job performance 
measuranent discussed were problems associated with the dimension- 
ality of performance criteria, their selection and evaluation, their 
predictability, their ultimacy, and the influence of environmental 
factors. 

It v?as concluded that there is need for an integrating con- 
ceptual framework to order and organize the field of measuring indi- 
vidual differences in on-the-job performance and to provide a more 
satisfactory ba<;is for evaluating on-the-job performance. — Auth, 

Seagren, P.W. Evaluation of VooationaUTeohnioal Education Program 
Effectiveness (Speech), Division of Vocational Education, 
University of California at Los Angeles, Calif., 19 August 
1969, 12 pp. 

Program management techniques were adopted from business and 
industry in order to provide an efficient and effective program of 
occupational training. The techniques include: (1) loss of tools 
or damage reports, (2) check-in system for instructors, (3) trainee 
attendance control (4) use of facilities, (5) organizational struc- 
ture, (6) instructor evaluation, (7) production requests, (8) curr- 
icular standards, (9) cost effectiveness, (10) advisory committees, 
and (11) accreditation. — C.H. 

Siegel, Arthur I., Schultz, Douglas G., and Federman, Philip. Post- 
Training Performance Criterion Development and Application: 
A Matrix Method for the Evaluation of Training^ prepared for 
Personnel and Training Branch, Office of Naval Research, by 
Applied Psychological Services, Wayne, Pa., January 1961. 

This report demonstrates the use of ••suitability" for the job 
as a basis for training evaluation by: (1) describing a specific 
scheme for quantitatively summarizing suitability, and (2) illus- 
trating the application of the scheme through data collected in pre- 
vious Applied Psychological Services' studies of four naval ratings. 
'♦Suitability'' for the job is defined as the training graduates' 
ability to do the tasks involved in the job. 

Matrix solutions are described that yield three indices, each 
reflecting a different aspect of the comparison between the skills 
of the trained man and the job's requirements. The method avoids 
the necessity for the determination of the functional relationship 
between training emphasis and job proficiency. Listing of the tasks 
as they are classified in the cells of a matrix and as they thereby 
contribute to the various training indices provides a further basis 
for consideration of specific changes in emphasis in various parts 
of the training program. The characteristics and limitations of the 
method are discussed and the results of its application to four naval 
ratings presented. — Auth. 
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Stufflebeam, Daniel L. "Toward a Science of Educational Evaluation," 
Educational Technology, Vol. 8, ♦No. 14, 1968, pp. 5-12. 

Suchman, Edward A. Evaluative Research: Principles and Practice in 
Public Service and Social Action Programs^ Russell Sage Founda- 
tion, New York, 1967, 186 pp. 

Edward A. Suchman, one of the world's foremost experts in 
measuring social behavior, presents the most comprehensive study of 
evaluation available to date. In thi^ report he describes the 
techniques used to determine empirically the extent to which social 
goals are actually being achieved, to locate the barriers to the 
achievement of these goals, and to discover the unanticipated con- 
sequences of social actions. 

This book will have many uses. It will aid the evaluative 
research person in striking a balance between rigorous method and 
the situation in which he must function. For the operating practi- 
tioner, the book will explain what competent evaluation involves. 
Administrators will find the volume an invaluable aid. — Auth* 

Sullivan, Howard J. "Objectives, Evaluation, and Improved Learner 
Achievement," in Popham, W.J., Eisner, E.W., Sullivan, H.J., 
and Tyler, L.L. , Instructional Objectives^ American Educa- 
tional Research Association Monograph on Curriculum Evalua- 
tion No. 3, Rand McNally, Chicago, 1969, pp. 65-99. 

Tracey, William R. Evaluating Training and Development Systems, 
American Management Association, Inc., 1968, 304 pp. 

This practical workbook will help every training manager eval- 
uate and improve his company's instructional program to achieve maxi- 
mum returns on investments in time, effort, and money. This book 
provides more than two hundred detailed analysis sheets that show 
the training director how to examine, evaluate, and revise each 
aspect of his program. Applicable for every type of training pro- 
gram--from nonexempt through supervisory to managerial --these check- 
lists enable him to plan and organize training activities geared to 
the needs of his company. 

Here are specific evaluation criteria and detailed suggestions 
for using them. This helpful guide shows how t^: 

• identify strengths and critical weaknesses in existing pro- 
grams 

• devise procedures to improve the system 

• determine the resources required for training 
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• initiate appropriate control devices 

• evaluate the instructor--his appearance and speech, his 
techniques, his ability to evoke student participation 

Informed, cost-conscious training directors will welcome this 
book as an invaluable aid in setting up and improving training pro- 
grams that will produce more profitable results. — Auth. 

Ullery, J. William. Pro j eat ABLEy Development and Evaluation of an 
Experimental Curviculwn for the Hew Quiney (Mass.) Vocational- 
Technical School. Management and Evaluation Plan for Instruc- 
tional Systems Development for Vocational-Technical Education, 
Technical Report 15, American Institutes for Research, Pitts- 
burgh, Pa., and Quincy Public Schools, Quincy, Mass., April 
1970. 

U.S. Department of the Navy. Fundamentals of Navy Curriculum Planning y 
NAVPERS 93510-1, Bureau of Naval Personnel, Washington, D.C., 
October 1968. 

This publication is one of a new series of education and train- 
ing manuals to be published by the Bureau of Naval Personnel. When 
completed, the series (NAVPERS 93510) will encompass all elements of 
instructional technology, course design, and school management and 
administration appropriate to Navy education and training. The first 
of the series to be distributed to Navy training activities and schools 
is the Handbook for Writing Learning Objectives, NAVPERS 93510-2, 
which should be studied by school administrators and instructors as 
an adjunct to this publication. 

The "instructional systems engineering approach" (or "systems 
approach") to course planning and design is emphasized in this publi- 
cation along with its application to the development of curriculum 
documentation for use in Navy schools. 

Other publications planned for this integrated series will in- 
clude revised manuals on testing and evaluation, school administra- 
tion, instructor training, in-service training, and new manuals on 
subjects such as utilization of instructional television and programmed 
instruction. 

Warren, Malcolm W. "Evaluating Training Actions," Chapter 8 in 

Training for Results: A Systems Approach to the Development 
of Human Resources in Industry, Addison-Wesley, Receding, Mass., 
1969, pp. 112-120. 
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Webb, Eugene J,, Campbell, Donald T,, Schwartz, Richard D., and 

Sechrest, Lee, Unobstrusive Measures: Nonreaotive Research 
in the Social Sciences^ Rand McNally § Co,, Chicago, 111., 
1966, 225 pp. 

Wheeler, E. A. "Economic Considerations for Industrial Training: A 
Study of Criteria for Controlling Training Costs," Training 
and Development Journal^ Vol, 23, No, 1, January 1969, pp. 
14-18. 

Wblman, Jean M., Campbell, Vincent N., Jung, Steven M,, and Richards, 
Jaroes M. A Comparative Study of Proprietary and Non^Propri- 
etary Vocational Training Programs — Volume Final Report 
AIR-22300-11/72-FR, prepared for Office of Education, U,S. 
Department of Health, Education and Welfare by the American 
Institutes for Research, Palo Alto, Calif,, November 1972, 

In order to formulate policy decisions regarding post « secondary 
vocational education more adequately, the U.S. Office of Education 
contracted with AIR to perform a survey of proprietary and non- 
proprietary vocational training programs in four selected occupational 
areas (office, health, computer, technical) and four metropolitan 
areas of the U.S. (Atlanta, Chicago, Rochester, San Francisco). The 
survey was oriented around three broad questions: 1) What are pro- 
prietary schools like, and how do they compare with non-proprietary 
schools offering similar training programs? 2) What are the students 
like who go to proprietary schools, and how do they compare to stu- 
dents who attend non-proprietax*y vocational schools? 3) What do 
students gain as a result of attending proprietary schools, and how 
do their gains compare to the gains recorded by students who attend 
non-proprietary schools? 

As part of the study, a brief review of the literature was pre- 
pared. Structured interviews and questionnaires were developed to 
survey the institutions, their students, and their alumni. All in- 
struments were pilot-tested and revised as a result- Student question- 
naires were administered by school staff members. Alumni question- 
naires were Trailed, with follow-up reminders. A telephone study of 500 
non-responding alumni (77% of whom were reached) revealed no marked 
differences from earlier respondents. All questionnaires were scored 
by National Computer Systems. — Auth. 

Wolman, Jean M, , Campbell, Vincent N., Jung, Steven M., and Richards, 
James M. A Comparative Study of Proprietary and Non-Proprie- 
tary Vocational Training Programs — Volume II: Appendices^ 
Final Report AIR-22300-11/72-FR, prepared for Office of Educa- 
tion, U.S. Department of Health, Education, and Welfare by the 
American Institutes for Research, Palo Alto, Calif., November 
1972. 
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SECTION IX 



MILITARY DOCUMENTS 



Each branch of the service has its own set of regulations govern- 
ing OJT--but the review of military documents was not restricted to 
regulations. Many closely related documents and manuals must be con- 
sidered in order to understand how OJT is actually conducted and how 
it interfaces with other personnel subsystems. In the Army, for 
example, it is not possible to understand the OJT system without say- 
ing something about the relationship to resident schools programs, 
unit training programs, proficiency testing programs, and so forth. 

A bi.ief description of the OJT program for each branch of the 

service in the United States is contained in another report in this 

series (Stephenson and Burkett, 1975).^ This section lists the most 

important references for the Department of Defense and each of the 
services (listed alphabetically) . 

BIBLIOGRAPHY 



U.S. DEPARTMENT OF DEFENSE 

Office of the Assistant Secretary of Defense (Manpower and Re- 
serve Affairs) . The TRANSITION Program. A Source of Manpomr 
for Industry, Washington, D.C., June 1972. 

Office of the Secretary of Defense. DOD Directive System: 
Quarterly Listing of Unclassified Issuance and Subject Index, 
Washington, D.C., Period Ending 31 March 1972. 

DOD Directive 1100.4, 20 August 1954, subj : "Guidance for Man- 
power Programs." 

DOD * 4.rective 1200.7, 2 July 1970, subj: "Screening the Ready 
Reserve." ASD(M5RA) 

DOD Directive 1215.9, 7 November 1969, subj: "Initial Active 
Duty for Training in Reserve Components." ASD(M&RA) 



See page 4 of this report. 
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DOD Directive 1304,14, 3 September 1970, subj : ''Award of Vari- 
able Reenlistment Bonus and Proficiency Pay for Enlisted Per- 
sonnel-" ASD(MSRA) 

DOD Directive 2000.10, 17 January 1972, subj: "Selection and 
Training of Security Assistance Personnel." ASD(ISA) 

DOD Instruction 1100.14, 2 October 1968, subj: "Food Service 
Education and Training Program," ASD(I5L) 

DOD Instruction 1145.2, 3 June 1965, subj: "Armed Forces Examin- 
ing and Entrance Stations Program Policy," ASD(M) 

DOD Instruction 1300.10, 23 October 1969, subj: "Enlisted Career 
Development and Grade Management Information Program." 
ASD(M5RA) 

DOD Instruction 1322.2, 30 March 1971, subj: "United States 
Armed Forces Institute." ASD(M5RA) 

DOD Instruction 1332.6, 3 July 1969, subj: "Career Counseling 
of Military Personnel." ASD(M5RA) 

DOD Instruction 1332.24, 19 July 1968, subj: "Project TRANSITION 
Certificate of Training." ASD(M§RA) 

DOD Instruction 4145.7, 26 October 1962, subj: "Maintenance of 
the Joint Storage and Materials Handling Manual (TM 743-200- 
NAVSANDA Pub 284-AFM 67-3-NAV MC 1101)." ASD(I5L) 

DOD Instruction 7730.31, 30 September 1965, subj: "Report on 
Officer and Enlisted Training Output." ASD(M) 



U.S. DEPARTMENT OF THE AIR FORCE 

Office of the Chief of Staff. The USAF Personnel Plan. Volume J. 
Personnel Management ObjeativeSy Washington, D.C., 1 July 1973. 

Office of the Chief of Staff. The USAF Personnel Plan. Volume 
II. Officer Structure (TOPLIHE), Washington, D.C., 1 April 
1975. 

Office of the Chief of Staff. The USAF Pers.mnel Plan. 
Volme^III. Aiivrnn Structure (TOPCAP), Washington, D.C. , 
\ May 1973. 
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Office of the Chief of Staff. The USAF Personnel Plan. Volume 

III. Aiman Structure Annexes^ Washington, D.C., 2 July 1973. 
(For Official Use Only.) 

Office of the Chief of Staff. The USAF Personnel Plan. Volume 

IV. Reserve Forces Personnel Structure (TOPREP)j Washington, 
D.C., 1 July 1973. 

Office of the Chief of Staff. The USAF Personnel Plan. Volume 

V. Civilian Structure (TOPIC) y Washington, D.C., 20 Nov 1974. 

Aerospace Defense Command. ^'World-Wide On-The-Job Training 
Conference, 5-7 May 1971, held at Ent AFB, Colo. 

Aerospace Defense ComTiand. '^World-Wide On-The-Job Training 
Conference, 21-24 August 1972," held at Ent AFB, Colo. 

Air Force Communications Service. **OJT Upgrade Training, 
Statistics,'* January through December 1971. 

Air Training Command. Technical Training. Planning ^ Preparation^ 
and Quality Control of Career Development Courses^ ATC Manual 
52-2, Randolph AFB, Texas, 1 October 1971. 

Air Training Command. Technical Training Pat^exms of Technical 
Training, ATC Pamphlet 52-1, Randolph AFB, Texas, 20 December 
1971. 

Air University. Extension Course Institute. ECI Catalog and 
Guide for Extension Course Administration (l9th edition), 
GunterAFS, Alabama, 1 October 1974. 

Air University. Extension Course Institute. Writing The USAF 
Extension Course, A Guide for Author^s (6th edition) , Gunter 
AFS, Alabama, January 1973. 



Air Force Manuals 

AFM 30-3 Volume II. Personnel, Mechanized Personnel Procedures, 
Advanced Personnel Data System (ADPS), Washington, D.C., 
10 September 1974. 

AFM 30-3 Volume VI. Personnel, Mechanized Personnel Procedures, 
Base Level Military Personnel Syetem, Washington, D.C., 
1 December 1972. 

AFM 35-1. Military Personnel. Military Personnel Classifi ja- 
tion Policy Manual (Officers, Warrant Officers, Airmen), 
Washington, D.C. , 18 August 1970 (Now AFR 35-1, 25 July 1974). 
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AFM 35-8. Military Personnel. Air Force Military Personnel 
Testing System^ Washington, D.C., 24 September 1971. 

AFM 39-1. Volume !• Enlisted Personnel. Airman Classification 
Manualy Washington, D.C., 29 December 1969. 

AFM 39-4. Enlisted Personnel. Airman Retraining /Laterial 
Training Programs:, Washington, D.C., 23 September 1971 (Now 
AFR 39-4^ 3 May 1974). 

AFM 39-11. Enlisted Personnel. Airman Assignment Mawialy 
Washington, D.C., 1 March 1973. 

AFM 50-2. Instructional System development^ Washington, D.C., 
31 December 1970. 

AFM 50-5. VSAF Formal School Catalog, Washington, D.C., 
1 September 1974. 

AFM 50-9. Principles and Techniques of Instruction^ Washington, 
D.C., 3 April 1967. 

AFM 50-23. Training. On-The-Job Training, Washington, D.C., 
15 August 1974. 

AFM 66-1. Volume I. Equipment Maintenance. Maintenance 
Management, Policy, Washington, D.C., 1 May 1974. 

AFM 66-1 Volume II. Equipment Maintenance. Chief of Maintenance 
(Aircraft and Missile) , Washington, D.C., 1 May 1974. 

AFM 66-1 Volume III. Equipment Maintenance. Organizational Mair- 
tenance, Washington, D.C., l May 1974. 

AFM 66-1 Volume IV. Equipment Maintenance. Field Maintenance, 
Washington, D.C., i May 1974. 

AFM 66-1 Volume V. Equipment Maintenance. Avionics Maintenance, 
Washington, D.C., 1 May 197^. 

AFM 66-1 Volume VI. Equipment Maintenance. Munitions Main- 
tenance, Washington, D.C., 1 May 1974. 

AFM 66-1 Volume VII. Equipment Maintenance. Consolidated 
Maintenance, Washington, D.C., l May 1974. 

AFM 66-1 Volume VIII. Equipment Maintenance. Airborne Missile 
Maintenance, Washington, D.C., l May 1974. 

AFM 66-1 Volume IX. Equipment Maintenance. Intercontinental 
Ballistic Missile Maintenance, Washington, D.C., i May 1974. 
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AFM 66-1 Volume X. Equipment Maintenanoe . Cormunicabions-EleC' 
tvonioS'-Meteorologioal Maintenance (Chief of Maintenance and 
Maintenance Activities)^ Washington, D.C., 1 May 1974. 

AFP 50-58, Volumes I-V. Handbook for Designers of Inatimetionat 
Systems, Washington, D,C., 15 July 1973. 

Air Force Regulations 

AFR 0-7. Indexes. Index of Air Force Perscnrtel Tests^ 
Washington, D.C., 16 December 1974. 

AFR 0-8. Indexes. HcmBrical Index of Specialty Training 
Standar'dsj Washington, D.C., 1 October 1974. 

AFR 8-13. Special Publications Systems. Air force Specialty 
Training Standards, Washington, D.C., 17 June 1974. 

AFR 39-6. Enlisted Personnel. Responsibilities of Noncom- 
missioned Officers (NCOs), Washington, D.C., 30 March 1972. 

AFR 39-29. Enlisted Personnel. Promotion of Airmen^ Washington, 
D.C., 1 August 1974. 

AFR 39-30. Enlisted Personnel. Administrative Demotion of 
Airmen^ Washington, D.C., 20 May 1974. 

AFR 50-9. Training. Special Training ^ Washington, D.C., 
6 March 1974. 

AFR 50-12. Training. Extension Course Program^ Washington, D.C., 
15 February 1974.^ 

AFR 50-38. Training. Field Evaluation of Formal School Graduates, 
Washington, D.C. , 5 December 1973. 

AFR 50-54. Training. Field Training Detachment (FTD) Program^ 
Washington, D.C, 30 November 1973. 



National Guard Bureau 

Air National Guard Manual. Training. On-the-Joh Training, ANGM 
50-23, Washington, D.C, 20 April 1972 (Now obsolete). 
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U.S. DEPARTMENT OF THE ARMY 
Army Field Manuals 

FM 21-5. Military Training Management^ Washington, D.C., 
18 December 1964. 

FM 21-6. Techniques of Military Instruction^ Washington, D.C., 
20 January 1967. 

Army Pamphlets 

Pam 310-8. Military PubLications. Index of Army Personnel Tests 
and Measures 9 Washington, D.C., April 1971. 

Pam 351-20. Announcement of Amy Correspondence Courses^ 
Washington, D.C., March 1972. 

Army Regulations 

AR 220-55. Field Organizations. Field and Command Post Exercises, 
Washington, D.C., 13 July 1973. 

AR 350-1. Training. Army Trai'iing^ Washington, D.C., 29 October 
1973. 

AR 350-27. Training. Skill Development Base^ Washington, D.C,, 
2 June 1969. 

AR 351-20. Schools. Army Correspondence Course Program^ 
Washington, D.C., 18 May 1973. 

AR 621-5. Education and Training. General Educational Develop- 
menty Washington, D.C., 26 August 1974. 

U.S. Army Training S Doctrine Command 

TRADOC Pam 350-11. Training. Systems Engineering of Unit Train- 
ingy Washington, D.C., 12 January 1973. 

TRADOC Pam 600-11. Personnel. Guidelines for the Conduct of 

Per formance-Oriented Training y Washington, D.C., 22 October 1973. 

TRADOC Reg 350-1 Anx B. Army Advanced Individual Training Policies 

and Administration y Washington, D.C., 15 August 1973 (AR Supple- 
ment) . 
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TRADOC Reg 350-1 Anx D. USAR Training Divisions and Separate 
Advanced Individual Training (AIT) Brigades Taaining Policies^ 
Washington, D.C., 1 August 1973 (AR Supplement). 

TRADOC Reg 350-1 Anx F. On-The-Job Training (OJT) in Units ^ 
Washington, D.C., 20 August 1973 ('\R Supplement). 

TRADOC Reg 350-43 Training. Liaison and Relationships e,. 
Service Schools and Training Centers Regarding Individual. 
Training, Washington, D.C., 1 February 1974. 

TRADOC Reg 350-100-1. Training. Systems Engineering of Training 
(Course Design), Washington, D.C., 20 April 1972. 

TRADOC Reg 351-3. Education and Training. TRADOC Schools Cur- 
riculurr Administration and Training Policies, Washington, D.C., , 
31 July 1973. 

TRADOC Reg 351-5. Schools. Designation of Military Occupa- 
tional Specialties (MOS) /Additional Skill i:entifiers (AST) 
Proponcncy , Washington, D.C., 8 October 1974. 

TRADOC Reg 614-200. Assignments , Details, and Transfers. 
Enlisted Personnel Selection, Tratning, and Assignment Sys- 
tem; Grades El through E9, Washington, D.C. , 1 July 1973 
(AR Supplement ) . 

U.S. Or.PARTMl-lNT OF TIIH iNAVY 

Office of the Chief of Naval Operations. Department of the Navy 
Glossary of Terms for Manpower Management and Personnel Admin- 
istration (first edition), OPNAV 01B1-P2, Washington, D.C, 
April 1968. 

Office of the Chief of Naval Operations. Glossary of Navy Train- 
ihj Tcvmlnoloay, OPNAV 37S56P2A, Washington, D.C, October 1969. 

Office of the Chief of Naval Operations. "Navy Training Policy," 
OPNAVINST 1500.18, Washington, D.C, 19 March 1973. 

Office of the Chief of Naval Operations. "General Military 
Training," OPNAVINST 1S00.22B, Washington, D.C, 23 April 
1973. 

Office of the Chief of Naval Operations. "Navy Correspondence 
Course Program," OPNAVINST ir>22.2A, Washington, D.C, 16 
Novenbcr 1971 . 
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Office of the Chief of Naval Operations, "Shipboard Training," 
OPNAVINST 3500. 32A, Washington, D.C., 17 January 1972. 

Bur Ndval Personnel. Handbook fo:" Writing Learning Objec- 

Ui.es, NAVPKRS 93510-2, Washington, D.C. , February 1968. 

Bureau of Naval Personnel. Manual of Navy Enlisted Classifica- 
tions, NAVPKRS 15105Z, Washington, D.C, January 1974. 

Bureau of Naval Personnel. Manual of Navy Officer Classifications, 
NAVPERS 15839B, Washington, D.C, Septt;Tnber 1968 (Change 8, 1974). 

Bureau of Naval Personnel. Manual of Qualifications for Advance- 
ment in Rating, NAVPERS 18068C, Washington, D.C, June 1971. 

Chief of Naval Hduc S Tng. Bibliography for Advancement, NAVEDTRA 
10052V, Washington, D.C, March 1974. 

Chief of Naval Educ S Tng. Educational Service Manual, NAVEDTRA 
15229, Washington, D.C, 1967. 

Chief of Naval Educ S Tng. List of Training Manuals and Cor- 
respondence Courses, NAVEDTRA 10061AH, Washington, D.C, 
March 1974. 

Chief of Naval Educ 5 Tng, Manual for Navy Instructors, NAVEDTRA 
16103C, Washington, D.C, 1964 (NAVEDTRA 107, 1975). 

Chief of Naval Educ Pi Tng. Quartermaster 3 and 2, Navy Training 
Course, NAVEDTRA 10149E, Washington, D.C, 1968. 

Chief of Naval Educ Tng. Quartermaster First and Chief, Rate 
Training Manual, NAVEDTRA 10151D, Washington, D.C, 1969. 

Naval Training Equipment Center. Index of Publications, NTEC 1480, 
Orlando, F-lorida, January 1973. 

Naval Training Equipment Center. Training Device Guide, NTEC 530-2, 
Orlando, Florida, July 1971. 

U.S. MARINE CORPS 

Headquarters U.S. Marine Corps. "Marine Corps Directives System 
Checklist (Issued as of 31 December 1974), Marine Corps 
Bulletin 5215, Washington, D.C, 10 January 1975 (Issued Quarterly). 

Headquarters U.S. Marine Corps. "Lateral Movement," Marine Corps 
Ordej ^??0,SB, Washington, D.C, 6 December 1973. 
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Headquarters U.S. Marine Corps. Marine Corps ^^^^-^f^^^ f^^^^ 
Qualification Training (Ground) ^Harine Corps Order PI 500. 32, 
Washington, D.C. , 13 June 1972. 

Headquarters U.S. Marine Corps, Individual Training of Enlisted 
Marines, Marine Corps Order 1510. 2H, Washington, D.C, 16 July 
1974. 

Headquarters U.S. Marine Corps. Unit Level Training Management » 
Marine Corps Order P1510.26, Washington, D.C, 4 April 1971. 
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SECTION X 
INNOVATIONS 



As the senior author reviewed the literature, some references 
attracted his attention in the sense that they suggested possible 
changes in Air Force policies and procedures. The idea, whatever 
it was, was written immediately and attached to the reference. 
Approximately 80 of these "innovation'' citations, based upon 40 
ideas, were identified. 

A list of innovations was prepared and many of them were dis- 
cussed during the headquarters and field-activity visits (de- 
scribed in Stephenson and Burkett, 1975),^ Some of the ideas did 
not survive criticism. Others were found worthy of further review. 

Since the literature review was frankly action and decision- 
oriented, it was not considered adequate to simply list various 
references next to the ideas with which they were associated. In- 
stead, one-page action-oriented sum.maries of what was involved 
in each idea were prepared for each possible innovation. A problem 
was briefly defined, the proposed solution was described, advantages 
were listed, and resource requirements were estimated. The litera- 
ture review thus took on decision-oriented aspects that went far 
beyond the references themselves. 

The list of possible innovations was organized into nine 
subject-matter categories: 

Administrative 
Audiovisual Presentations 
Computer-Assisted Techniques 
Evaluation 

Incentives -Motivation of Trainees 
Instructional Techniques 
Periodic Surveys 
Program Design 
Other 

Each innovation is described and presented here. Additional 
ideas (inspired by field-activity visits and the analysis of survey 
data) arc contained in Stephenson and Burkett (1975). 



^See page 4, this reporc. 
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ADftirilSTPATIVE 



INNOVATION 
PROBLEM 



SOLUTION 



ADVANTAGES 



RESOURCE 
REQUIREMENTS 



COMMENT 



Redefine Objectives for Remedial Programs and Reading 
Improvement Training 

Most remedial reading programs are now administered at 
field-activity locations. Trainee progress is delayed, 
because time must be spent away from the job to parti- 
cipate in the reading program. Upgrade training and 
work performance suffer as a result of these trainees 
who cannot read the materials required to perform their 
jobs. Present reading improvement programs are not stan- 
dardized across commands, and the training is often con- 
tracted out to local school disctricts. Objectives of 
these programs are not always realistic, in tl.at they 
attempt to improve general rather than job-related 
reading skills. 

Redefine objectives for remedial reading programs by 
adopting a functional literacy approach designed to de- 
velop required minimum job reading skills. These job 
reading programs should be built around clusters of re- 
lated specialties having a high incidence of problem 
readers. 

More effectiveness in improving needed job-specific 
readng skills; remedial objectives which are more 
realistic and attainable; cost-effectiveness associated 
with a standardized program geared to job and OJT re- 
quirements. 

It is estimated that 10% of the trainees need such crain- 
ing, and that the average training time required is four 
to twelve weeks. Program development costs are estimated 
to be about 3 man-years for the minimal clusters required. 

It is necessary to define Air Force objectives for reading 
programs very carefully. One should not expect to solve, 
in a short time, those individual general literacy problems 
that community school programs have been unable to solve 
in years of education. The more limited objective of 
improving job-related reading skills appears to be more 
realistic and cos^c-effective. 




SELECTED Duncan, Ross. Project One Hundred Thousand: Help for the 
REFERENCES Under-^Educated Adults The George Washington University 

School of nducation (Unpublished paper, 7 December 1970). 

(ASDIRS 3523) 
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Sticht, Thomas G. (Principal Investigator) and Cay lor, 
J. S. (Associate Investigator). '^Development of Proto- 
type Job-Functional Army Literacy Training Program 
(FLIT),'*- in Work Unit Summaries, Defense Documentation 
Center, Defense Supply Agency, Alexandria, Va., DDC 
Report No. CT9150, 22 August 1972, p. 19. (Performing 
Organization: Human Resources Research Organization, 
HumRRO Division No, 3, Presidio of Monterey, Calif.; 
Contract No. DAHC19-70-C-0012.) 

Sticht, Thomas G. (Ed.). Reading for Working: A Funo^ 
tional Literoay Anthology ^ Human Resources Research 
Organization, Alexandria, Va. , 1975. 

Sticht, Thomas G., et al . HimRRO's Literacy Research for 
the U.S. Army: Developing Functional Literacy Training, 
HumRRO Professional Paper 13-17, Human Resources Research 
Organization, Alexandria, Va, , December 1973. 

This paper summarizes literacy research and development 
perfonned by HumRRO for the Army since 1968. Literacy 
needs for several basic Army Military Occupational Spe- 
cialties (MOSs) were identified and methodology was 
developed to evaluate reading requirements for Army jobs. 
Under the current effort, an experimental training pro- 
gram is being designed to produce a level of functional 
literacy appropriate to minimal MOS requirements. — Auth. 

Vineberg, Robert, Sticht, Thomas G. , Taylor, Elaine, 
and Caylor, John S. Effects of Aptitude (AFQT), Job 
Experience, and Literacy on Job Performance: Sivmary 
of HumRRO Work Units UTILITY and REALISTIC, HumRRO Tech- 
nical Report 71-1, Huma^^ Resources Research Organization, 
HumRRO Division No. 3, Presidio of Monterey, Calif,, 
February 1971 . 

(f-or abstiact see p. 124.) 
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ADMINISTRATIVE 



INNOVATION Rotation Plans for Selected Specialties When the 

Variety of Training at One Work Center Is Inadequate 

PROBLEM Circumstances such as the variety of trainirg available 

at a given location and unusual production pressures 
often restrict the training that is available to train- 
ees. These restrictions are acceptable in the dual- 
training concept, since it is possible to be upgraded 
based upon learning a single task in a specialty. If 
this requirement is changed (as suggested in another 
innovation in this list), some kind of rotation plan 
must be formally implemented. 

SOLUTION Require rotation plans, when the variety of training 

that is available at a given work center is not adequate 
to meet minimum needs; these plans should be flexible 
(e.g., three tasks out of a list of ten must be learned) 
for selected specialties. 

ADVANTAGES Personnel in a specialty would be qualified to perform 
a reasonable variety of tasks before being upgraded. 
Morale problems because or an inadquate variety of 
training tasks would be less likely to occur. 

Once the flexible, minimum standards have been deter- 
mined and the Specialty Training Standard (STS) has 
been modified, the time required to establish a 
rotation plan is relatively trivial--a committee of 
NCOS might have to meet once every six months for a 
few hours. 

SELECTED Christal, Raymond E. (Principal Investigator). ^Troce- 

REFERENCES dures for Optimal Career Development and Talent Utiliza- 
tion," in Work Unit StanmaHeBy Defense Documentation Cen- 
ter, Defense Supply Agency, Alexandria, Va. , DDC Report 
No. CT9150, 22 August 1972, p. 61. (Performing Organi- 
zation: Air Force ^^aman Resources Laboratory, Lackland 
AFB, Texas.) 

Tuttle, T. C. (Principal Investigator). "Development 
of Improved Air Force Assignment Systems," in Work 
Unit Surmaries, Defense Documentation Center, Defense 
Supply Agency, Alexandria, Va., DDC Report No. CT9150, 
22 August 1972, p. 62. (Performing Organization: Air 
Force Human Resources Laboratory, Lackland AFB, Texas.) 



RESOURCE 
REQUIREMENTS 
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U.S. Departinent of Labor, Manpower Administration. To- 
ward the Ideal Journeyman. Volume 1. An Optimum Train- 
ing System in Apprentiaeable Occupations, Manpower 
Research Monograph No. 20, Manpower Administration, 
Washington, D.C., 1970. 
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ADMINISTRATIVE 



INNOVATION Periodic Comparisons of Technical School and OJT Grad- 
uates in Order to Facilitate Decisions About Cost 
Effective Combination of Technical School and OJT for 
Category B Specialties 

PROBLEM Category B (semiskilled) specialties in the Air Force 

can be taught either in school or on the job (OJT). 
Cost factors determine the number who only receive OJT. 
The decisions are difficult to make, since some of the 
costs are difficult to quantify, and no one model has 
been found acceptable by everyone. The situation is 
further complicated, because studies of the relative 
costs of OJT and formal schools training have produced 
inconsistent results; and the policies are somewhat 
controversial within the Air Force. 

SOLUTION Studies ccwnparing OJT graduates and technical school 

graduates with respect to the time required after basic 
training before the 5- level is awarded can be conducted 
on a regular basis. If the advantage is in favor of 
the OJT graduates, more use of OJT can be recommended; 
if it favors the technical school graduates, less use 
of OJT can be recommended. 

Cost savings associated with improved decisions about 
the combination of OJT and resident school training 
(e.g., shorter training time for upgrade trainees). 

Reports of this type could be periodically provided for 
all Category B specialties. Approximately one man-year 
of effort would be required. 

Time to complete upgrade training is not so reliable an 
index as one would hope. The basic problem is that al- 
most everyone finishes upgrade training in the minimum 
amount of time. The problem is complicated by the pro- 
cedures by which personnel are selected for OJT and how 
they vary from specialty to specialty. Nevertheless, 
the information should be considered, when decisions 
are being made about an optimum combination of OJT and 
resident school training. 

SELECTED Black, D. and Bottenberg, R. A. Comparison of Tech- 

REFERENCES nioal School and On-'The-Job Training as Methods of 



ADVANTAGES 



RESOURCE 
REQUIREMENTS 



COMMENT 
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skill Upgrading^ AFHRL-TR-70-48, Personnel Division, 
Air Force Human Resources Laboratory, Lackland AFB, 
Texas, Decesber 1970. (AD 7^' 530) 



(For abstract, see Section V, References 0 
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INNOVATION 



PROBLEM 



SOLUTION 



ADVANTAGES 



RESOURCE 
REQUIREMENTS 



SELECTED 
REFERENCES 



Decision Tables to Facilitate Decisions About When 
Someone Should be Decertified in a Previously Learned 
Skill Because of Degradation of Performance 

The decision as to whether someone should be decerti- 
fied in a certain STS category is difficult to make. If 
people are decertified unnecessarily, there is much 
wasted time, while such persons retrace their steps to 
learn the task. If people who should be decertified are 
not decertified, there is a real danger of accidents 
that could damage personnel or equipment. 

Design decision tables that will facilitate decisions as 
regards when someone should be decertified in a previ- 
ously learned skill. 

Additional guidance to those who must make decisions as 
regards people being qualified or not; improved job 
performance and safety. 

Research is needed to design and evaluate ways of pre- 
dicting degradation of performance, and to develop a 
decision table that would have proven utility. More 
research would then be needed to field test the effec- 
tiveness of the decision tables before they could be 
incorpurated into Air Force manuals. 

Sitterley, T. K. and Berge, W. A. (The Boeing Company, 
Seattle, Washington). Degradation of Learned Sktlls: 
Effeativeness of Practice Methods on Simulated Space 
Flight Skill Retention, Report D180- 15081-1, prepared 
for Manned Spacecraft Center, National Aeronautics and 
Space Administration, July 1972. 

Manual flight control and emergency procedure task skill 
degradation was evaluated after time intervals of from 
one to six months. The tasks were associated with a 
simulated launch through orbit insertion flightphase of 
a space vehicle. The results showed that acceptable 
flight-control performance was retained for two months, 
rapidly deteriorating thereafter by a factor of 1.7 to 
3.1 depending on the performance measure used. Pro- 
cedural task performance showed unacceptable degradation 
after only one month, and exceeded an ordei- of magnitude 
after four months. The effectiveness of static re- 
hearsal (checklists and briefings) and dynamic warm-up 
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(simulator practice) retraining methods were compared 
for the two tasks. In general, static rehearsal 
effectively countered procedural skill degradation 
while some combination of dynamic warm-up appeared 
necessary for flight-control skill retention. Further, 
it was apparent that these differences between methods 
were not solely a function of task type or retraining 
method, but were a function of the performance measures 
used for each task, — Auth. 
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AUDIOVISUAL PRESENTATIONS 



INNOVATION Conversion of Selected CDCs Into Self-Paced Slide-Tape 
or Cassette-Film Presentations 

PROBLEM Marginal readers in some career fields have considerable 

difficulty completing conventional CDCs, 

SOLUTION Slide-tape cassettes are available and can be operated 

by inserting a single cartridge into an automatic slide- 
tape playing console about the size of a small TV set . 
When removed, the cartridge rewinds itself and is ready 
for the next user. Many programs of this type have been 
found to be • ffective with marginal readers because they 

use audio visual techniques to reduce reading demands, 

ADVANTAGES Greater interest on the part of trainees; more effective 
use of training time; improved visualization of equip- 
ment and procedures. 

RESOURCE Cassette film tapes cost approximately $80 apiece. The 

REQUIREMENTS viewing and playback apparatus cost approximately $400 
apiece, 

COMMENT The expense of using this type of equipment could be 

justified only under circvunstances where equipment, per- 
sonnel, or procedures require extensive visualization, 

SELECTED Baker, G,E, "Experiment in Teaching One-Point Per- 

REFERENCES spective," Industrial Arts and Vocational Education, 
Vol. 45, May 1966, pp, 46-47, 

Bumap, E.G, "On-The-Job Instruction with Programmed 
Tapes," Training Directors Journal^ Vol, 19, No. 10, 
1965, pp. 42-48. 

Crowder, Gene A, "Visual Slides and Assembly Models 
Compared With Conventional Methods in Teaching In- 
dustrial Arts" (Doctoral thesis), Texas ASM University, 
College Station, Texas, 1968. 

Plug, Eugene R, F, "An Experimental Evaluation of 
Selected Presentation Modes in the Self-Instruction of 
a Manipulative Industrial Arts Learning Task," (Doctoral 
thesis). University of Minnesota, Minneapolis, 1967. 

This report summarily describes a study to evaluate che 
utilization ot inflight audio-video recording (AVR) and 
ground playback equipment in the United States Air Force 
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T-37 Pilot Instructor Training (PIT) Program. It in- 
cludes a description of the preparation for, and conduct 
of, a demonstration designed to permit comparisons be- 
tween Instructor Pilot (IP) t-«^ainees who used the AVR 
system during their training (TV group) and those who did 
not (non-TV group). The design included the development 
and use of (a) special grading sheets for recording eval- 
uations of IP trainee pilot and instructional performance 
on six maneuvers; (b) a special check-ride to provide ad- 
ditional information on the capability of a graduating 
trainee to analyze pilot performance errors and apply 
the principles of instruction in presenting corrective 
measures to Undergraduate Pilot Training (UPT) students; 
and (c) a questionnaire designed to solicit effective and 
ineffective instructional situations encountered by new 
and experienced UPT IPs. Operational considerations 
limited the conduct of the study as originally designed, 
resulting in a less than ideal niMber of subjects re- 
ceiving the TV treatment. The results are described in 
terms of (a) the differences in performance levels be- 
tween the TV and non-TV group of trainees (no difference), 
(b) an analysis of the attitudinal comments made by the 
IPs and IP trainees who used the AVR system regarding 
its application as a training aid (favorable and en- 
thusiastic), (c) suggested changes to the hardware de- 
sign of the system, and (d) recommendations for system 
procurement and implementation into the PIT program and 
other pilot training programs (conditionally recommended). 

Homer, W.R. and Shettel, H.H. Audio/Video Recording 
in Pilot InstTUotov Training^ Flying Training Division, 
Air Force Human Resources Laboratory, WilliaiUS AFB, 
Arizona, February 1972. 

Moeller, C.A. "A Comparison of Selected Audio-Visual 
Methods and Lecture Demonstration Methods in Teaching 
Manipulative Skills Related to Metal Operations," 

Journal of Industrial Teacher Education^ Vol. 4, March 
1967, pp. 20-29. 

North, Stafford (Oklahoma Christian College). "Ivy 
Walls, Tapes, and Workbooks/' in Papers Presented at a 
Symposium on Programmed Learning Systems^ held at the 
Christopher Inn, Columbus, Ohio, 18 May 1965, sponsored 
by North Electric Co., Calion, Ohio. 
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Pieper, W.J., Catrow, E. J. , aiid Swezey, R,W, (Applied 
Science Associates, Inc.), and Smith, E.A. (Air Force 
Human Resources Laboratory, Lowry AFB, Colo.)* "Auto- 
mated Apprenticeship Training (AAT) : A Systematized 
Audiovisual Approach to Self-Paced Job Training,'* Cata- 
log of Selected Documents in Psychology , a publication 
of the Journal Supplement Abstract Service (MS No, 315), 
Vol. 3, Winter 1973, p. 21. (See also Air Force Human 
Resources Ladoratory Technical Report, AFHRL-TR-72-20.) 

Two Automated Apprenticeship Training (AAT) courses were 
developed, administered, and evaluated for Air Force 
Security Police Law Enforcement and Security specialists. 
AAT is a systematized audio- visual approach to self- 
paced job training which employs an easily operated, 
portable and reliable teaching device. AAT courses were 
developed to be job specific and were based on a behav- 
ioral task analysis of the two Security Police specialty 
areas. AAT graduates were compared with graduates of 
comparable Airman Basic Resident (ABR) course and Career 
Development Course (CDC) for the same jobs in a Training 
Regime by Aptitude Group design. Evaluation criteria 
included a job specific performance test, an apprentice 
knowledge test and supervisor's ratings. Results in- 
dicated superior scores for the AAT graduates on the job 
performance test, and no differences among Training 
Regimes on the other criteria. A sigiiificant Aptitude 
effect was also obtained on the job performance test. 
The AAT Course was considered superior to other Training 
Regimes in terras of man-hours expended. Training super- 
visors also expressed a preference for the AAT technique. 
--Auth. 

Scharf, George P. "Modified Fire Protection Specialist 
Career Development Course for New Mental Standards Per- 
sonnel,*' paper presented at the 15th Annual Conference, 
Military Testing Association, held in San Antonio, Texas, 
28 October-2 November 1973. 
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AUDIOVISUAL PRESENTATIONS 



INNOVATION Learning by Listening 

PROBLEM Some Air Force personnel are functionally illiterate, 

but they can understand the spoken word relatively well. 
Tape recordings of lessons seem to work better for such 
people than do CDC volumes. 

SOLUTION Tape-record CDCs and make them available to personnel 

in selected specialties who prefer or need such a mode 
of instruction. 

ADVANTAGES Air Force personnel who have difficulty reading can 

still make progress in their CDCs; Air Force personnel 
who prefer to listen rather than read can do so. 

Dcn^i'^^-^-- The least expensive method is to simply tape-record the 
REQUIREHENTS text, and send the text along with the cassette to those 
who prefer to read ^ni listen. 



COMMENT 



Control of tape recorders is likely to be a problem. 



DcccDlM?rc ^urTaH , Thomas E. (Navy Training Research Laboratory) 
REFERENCES and Brock, John F. (Fleet Anti-Air Warfare Training 

Center). Programed Instruction for Selected CIC Watch 
Officer Tasks: I. An Experimental Evaluation of the 
Audio notebook in the Teaching of Radiotelephone ^ SRR 
68-11, U.S. Naval Personnel Research Activity, San Diego, 
Calif., November 1967. (AD 664 235) 

Scharf, George P. ^'Modified Fire Protectioi Specialist 
Career Development Course for New Mental Standards Per- 
sonnel," paper presented at the 15th Annual Conference, 
Military Testing Association, held in San Antonio, Texas, 
28 October-2 November 197.3. 

Sticht, Thomas G. Learning by Listening in Relation to 
Aptitude^ Reading^ and Rate-Controlled Speech^ HumRRO 
Technical Report 69-23, Human Resources Research Organi- 
zation, Division No. 3, Presidio of Monterey, Calif. 
December 1969. 

A series of studies was performed to explore the pos- 
sibility of substituting listening for reading require- 
ments, with special reference to marginally literate 
Category IV personnel. Time-compressed speech was eval- 
uated as a means of producing listening rates comparable 

ERiC i^.^ 



to silent reading rates. The results indicated that for 
both average and low aptitude men, listening was as ef- 
fective as reading for obtaining factual information from 
test passages varying in difficulty level. Both high 
and low aptitude men performed more efficiently with 
moderate (36%) amounts of time compression than with no 
compression of the listening selections. Additional 
evaluations of time-compressed speech were made, and ed- 
ucation and training implications of the research were 
discussed, — Auth, 

Sticht, Thomas G, Learning by Listening in Relation to Apti- 
tudey Reading and Rate-Controlled Speech: Additional Studies, 
IfumRRO iechnical Report 71-5, Human Resources Research 
Organization, Division No, 3, Presidio of Ilontery, Calif., 
April 1971. 

A series of experiments explored the feasibility of 
substituting listening for reading requirements in Army 
training and jobs, with special reference to marginally 
literate, AFQT Mental Category IV men. Results of these 
experiments and related earlier research are summarized. 
Major findings indicate that high and low aptitude men 
may learn certain materials as well by listening as by 
reading; some poorer readers prefer to learn by listening 
rather than by reading. Characteristics of the recorded 
message that were found to affect listening comprehension 
include difficulty level of message, linguistic features 
of speech, and rate of speech. Extensive studies of the 
ase of time- compressed and expanded recordings are de- 
scribed, — Auth* 
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AUDIOVISUAL PRESENTATIONS 



INNOVATION Decision Flow Charts for Selecting Communications Media 

PROBLEM There are so man/ different media, and so many fads in 

training program design, that it is difficult to decide 
which media would be best under which circumstances. 

SOLUTION Decision flow charts for selecting communications media 

for each part of the training process and for different 
kinds of training programs, 

ADVANTAGES Expert decisions about communications media can be made 
quickly by personnel who are not well-read in the field, 

RESOURCE Some decision tables have already been designed, but 

REQUIREMENTS they have not been widely dispersed to training design- 
ers. The use of these decision tables needs to be 
evaluated, and revised decision tables need to be de- 
signed. 

SELECTED Bretz, R. The Selection of Appropriate Communications 

REFERENCES Media for Instruction: A Guide for Designers of Air 

Force Technical Training Programs^ Rand Report R-601-PR, 
Rand, Santa Monica, Calif., February 1971. 

This report presents a description and discussion of the 
uses of communication media of all classes in instruc- 
tion. Communication media are systems that transmit 
messages for larger user systems which serve such pur- 
poses as instruction, information, entertainment, or 
propaganda dissemination. The function of a communica- 
tion medium is simply to communicate, and this function 
is the focus of this Report. No attempt is made to con- 
sider the values of communication, its results, or its 
effectiveness, as any such evaluation must be done in 
the specific context of the system being served by the 
communication medium. 

Eleven uses for communication in instruction are des- 
cribed, each of which has distinct requirements in 
terms of communication media, equipment configurations, 
and program content: 
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(1) Providing the learner with knowledge of his 
learning objectives; 

(2) Motivating the learner; 

(3) . Presenting information; 

(4) Stimulating discussion; 

(5) Directing learner activities; 

(6) Conducting drill and practice; 

(7) Reinforcing learning; 

(8) Providing a learner/simulator interface; 

(9) Evaluating learner progress and program 
effectiveness ; 

(10) Assisting in the administration of instructional 

systems; and 

(11) Assisting in research and development. 

The selection of the appropriate laedia for each of these 
uses is discussed, and criteria are given for determin- 
ing the need for various system capabilities in illustra- 
tive instructional situations. — Auth. 
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COMPUTER ASSISTED TECHNIQUES 



INNOVATION Computer Assisted Text Handling Systems for Rapid 
Revisions of CDC Volumes 

PROBLEM A complaint about CDCs is that some of them are out of 

date. One reason is that relatively minor revisions 
and changes could require retyping, editing, and proof- 
reading of whole volumes. 

SOLUTION Put CDC volumes on tapes, and let one of the recently 

developed text-handling subsystems make corrections and 
produce master printouts suitable for printing. 

ADVANTAGES More rapid updating of CDC volumes. 



RESOURCE 
REQUIREMENTS 



SELECTED 
REFERENCES 



It takes much longer and is more expensive to get text 
on the computer initially; but the costs of rewriting 
and editing are much less after the initial investment. 

Caylor, John S. , Sticht, Thomas G. , Fox, Lynn, and 
Ford, J. Patrick. Methodology for Evaluating Reading 
Requirements of Army JobSj HumRRO Technical Report 73-5, 
Human Resources Research Organization, HumRRO Division 
No. 3, Presidio of Monterey, Calif., March 1973. 

IMPACT Staff. Project IMPACT^-Computer-AdmCnistered 
Instruction: Description of the Hardwax'e/ Software Sub- 
systam^ HumRRO Technical Report 70-22, Human Resources 
Research Organization, HumRRO Division No. 1, Alexandria, 
Va. , December 1970. 

IMPACT Staff. Project BiP ACT —Computer ^Administered 
Instruction: Preparing and Managing the Content of In-- 
struation^ IMPACT Text^Handling Subsystem^ HumRRO 
Technical Report 71-21, Human Resources Research Organi- 
zation, HumRRO Division No. 1, Alexandria, Va., September 
1971. 



Project IMPACT is a comprehensive advanced development 
project designed to produce an effective and economical 
computer- administered instruction (CAI) system for the 
Army. This report describes the concepts, approach, 
and implementation of the Project IMPACT text-handling 
subsystem. The computer-based facilities for preparing, 
storing, and retrieving the content of CAI courses o£ 
instruction are described, as are CAI courses. Computer 
software tools are described in terms of their use by 
course authors. ---Auth. 
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Vineberg, Robert, Sticht, Thomas G. , Taylor, Elaine, 
and Cay lor, John S. Effects of Aptitude (AFQT)^ Job 
Experience, and Literacy on Job Performance: Summary 
of HumRRO Work Units UTILITY and REALISTIC, HumRRO Tech- 
nical Report 71-1, Human Resources Research Organization, 
HumRRO Division No. 3, Presidio of Monterey, Calif., 
February 1971. 

A series of studies were conducted to determine how Army 
personnel in Mental Category IV and in other mental 
categories compare in their job performance and in their 
overall suitability for military service. Information 
is provided concerning the demands for reading, arith- 
metic, and listening skills in four major military oc- 
cupational specialties. The performance of approximately 
1800 men with Army experience ranging up to 20 years was 
measured by intensive job sample tests, job knowledge 
tests, and superxdsor ratings. Inforitiation about back- 
ground, personal characteristics, and military experi- 
ences was obtained through biographical questionnaires, 
a battery of published and experimental tests, and Army 
records. The inajor findings and conclusions are given 
in this summary report, which will be followed by several 
detailed reports on various research phases. --i4Mt?z. 
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COMPUTER ASSISTED TECHNIQUES 



INNOVATION Screen End-of-Course Examinations and Specialty Knowl- 
edge Tests (SKTs) for Reading Level by Using Computer- 
ized Reading-Level Evaluation Procedures 

PROBLEM The language in which a test is written can make it 

unnecessarily difficult to pass for someone who is a 
member of an underprivileged minority. Long words, or 
words that one is not likely to encounter unless one is 
part of the American middle class, sometimes get into 
EOCs and SKTs, As a result, some training examinations 
seem to resemble vocabulary tests more than they re- 
semble tests of job knowledge, 

SOLUTION Procedures are being designed by which a computer can 

quickly indicate the reading level of sample test items. 
The items are keypunched on cards, and submitted to the 
computer for analysis. The computer determines word and 
sentence length, checks the word frequency in the popula- 
tion at large, and provides a set of reading- level scores. 
Cutting scores would prohibit the items from being used, 
if the scores fell below certain points, 

ADVANTAGES Fairness in tests; and improved equity in decisions 
based upon the test performance, 

RESOURCE Special procedures would be needed for dealing with 

REQUIREMENTS technical jargon before any automated reading-level in- 
dex could be used. The development of these procedures 
requires further research. Once the procedures have 
been designed, however, resource requirements (key- 
punching, computer tirp.?) v;ould be minimal, 

SELECTED Caylor, John S, , Sticht, Thomas G, , Fox, Lynn, and Ford, 

REFERENCES J, Patrick, Methodology for Evaluating Reading Require- 
ments of Army Jobs^ HumRRO Technical Report 73-5, Human 
Resources Research Organization, HumRRO Division No, 3, 
Presidio of Monterey, Calif,, March 1973. 

Van Doren, R, (ed,), *'Bread and Fire: Manpower, Human 
Assessment and tlie Disadvantaged,** American Vocational 
Joumalj Vol, 48, No, 1, 1973, pp, 85-100, 

Vineberg, Robert, Sticht, Thomas G,, Taylor, Elaine, and 
Caylor, John S, Effects of Aptitude (AFQT)^ Job Experi- 
ence^ and Literacy on Job Performance: Summary of HumRRO 
Work Units UTILITY and REALISTIC^ HumRRO Technical Re- 
port 71-1, Human Resources Research Organization, HumRRO 
Division No, 3, Presidio of Monterey, Calif, , February 
1971, 
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COMPUTER ASSISTED TECHNIQUES 



INNOVATION 



PROBLEM 



SOLUTION 



ADVANTAGES 



Computer-Assisted Item Writing and Test Assembly 
Procedxares 

CDC end-of-course tests must be available in several 
different versions, and must be revised periodically. 
Otherwise, the test content would become generally 
known, and personnel who have already taken a test 
could be in a position to help others to be prepared 
for it. 

Computer-assisted item writing, techniques based upon 
text scanning procedures, and test assembly procedures 
based xxpon a common pool of items that are categorized 
by subject matter area for rapid updating and revision. 



Test development should be more economical once a 
satisfactory set of procedures has been developed, 
would be more tests available at less cost. 



There 



RESOURCE 
REQUIREMENTS 



SELECTED 
REFERENCES 



It is relatively easy to set up a computer-assisted test 
assembly technique, but a generally useful set of item- 
writing techniques has net yet been developed. The 
initial resource requirements for computerized test as- 
sembly techniques are estimated at one man-month per 
test, but savings in test development costs should more 
than compensate for this expense. Research and develop- 
ment costs for a satisfactory i m-writing technique are 
estimated at five man-years. 

Anastasio, E.J., et al . Computer-Assisted Item Writing^- 
IL Sentence Completion Items^ TDM-69-1, Educational 
Testing Services, Princeton, N.J., 1969. 

Carver, Ronald P. Procedures for Constructing a Variety 
of Information Processing Measures Appropriate for Pro- 
duction Materials^ Revrac Publications, Silver Spring, 
Md., 1971. 

Finucane, J.L. Item Banks: Developing Evaluation 
TestS'-Automated Process (DETAP)^ Technical Memorandum 
34, U.S. Army Enlisted Evaluation Center, Indianapolis, 
Indiana, May 1970. 
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EVALUATION 



INNOVATION Use of Trainee Confidence Ratings as a Diagnostic Aid 
for Evaluating Progress in Training 

PROBLEM Very little guidance is provided to OJT supervisor/ 

trainers regarding how they should go about evaluating 
the hands-on performance of OJT trainees in order to 
decide whether they are really qualified to perform a 
task. As a result, some trainees are undertrained and 
others are overtrained. 

SOLUTION Have the trainee rate his confidence in his ability to 

perform each task on his JPG before the supervisor/ 
trainer certifies that his training is completed. Per- 
mit the trainee to indicate whether he considers himself 
to be undertrained or overtrained in each category of 
training. File this information in his training record 
until a new JPG is prepared. 

ADVANTAGES More informed and more accurate evaluations of progress 
in training. 

RESOURCE Souie new forms would have to be designed. A field test 

REQUIREMENTS is needed before the procedures could be adopted. 

CO^WENT Personality factors (self-confidence) complicate the 

confidence ratings, as well as the overtraining and 
undertraining ratings. One can also argue, however, 
that the trainee who lacks confidence should be given 
more training than one who does not. 

SELECTED Echternacht, Gary J. Use of Confidence Testing in 
REFERENCES Objective Teata, AFHRL-TR-71-32, Technical Training 

Division, Air Force Human Resources Laboratory, Lowry 

AFB, Colo., July 1971. AD-734 031. 

The development of confidence testing as a form of 
objective testing was traced from Hevner's initial for- 
mat to that developed in recent years. Confidence test- 
ing has been used in varying forms over the past forty 
years as a method for increasing the amount of informa- 
tion available from objective test items. This paper 
traces the development of the procedure from Hevner's 
beginning method up to the various methods in use today. 
The term confidence testing is applied to both proba- 
bilistic testing and confidence weighting procedures. 
Various procedures are presented, and their relationship 
with personality factors is discussed. --Auth. 
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Echternacht, Gary J., Boldt, Robert F. , and Sellman, 
Wayne S. User's Handbook for Confidence Testing as 
a Diagnostic Aid in Technical Training^ AFHRL-TR-71-34, 
Technical Training Division, Air Force Human Resources 
Laboratory, Lowry AFB, Colo., July 1971. AD-731 192. 

This handbook is intended to supply both testing special- 
ists and general users of tests with a set of instruc- 
tions for implementing a program of confidence testing 
in technical training situations, provide information 
concerning such factors as the identification of promis- 
ing areas of application, the relative value and ease of 
alternative scoring methods, techniques for evaluating 
confidence information, and administrative consider at ions^ 
It contains a discussion of Pick -One and Distribute 100 
Points confidence formats, other confidence procedures, 
and the relative merits of each method, selection and 
confidence test scale scores, uses of confidence testing, 
and instructions for those administering confidence 
tests where either hand or machine scoring is used.-->lwtfi. 

Echternacht, Gary J., Sellman, Wayne S. , Boldt, Robert 
F. , and Young, Joseph D. An Evaluation of the Feasibility 
of Confidence Testing as a Diagnostic Aid in Technical 
Training, AFHRL-TR-71-33, Technical Training Division, 
Air Force Human Resources Laboratory, Lowry AFB, Colo., 
July 1971. AD-734 032. 

This report describes a study to determine the feasibil- 
ity and the cost-effectiveness of using confidence test- 
ing as a diagnostic aid in technical training programs. 
Two types of confidence testing, Pick-One and Distribute 
100 Points, were developed for comparison to conventional 
multiple- choice testing. The study was carried out in 
two technical training courses. Aerospace Ground Equip- 
ment Repairman (AGE) and Jet Engine Mechanic (JEM) , cur- 
rently being taught at Chanute Air Force Base, Illinois. 
The criteria for feasibility included end of block exam- 
ination grades J number of student remediational sessions, 
and both student and instructor attitudes. In addition, 
the relationship of various personality variables to 
confidence test scores was examined for both types of 
confidence testing. The major finding was that while 
scoring was somewhat more time consuming, end of block 
examination grades improved slightly and the number of 
remediations required declined slightly when either 
confidence testing method was employed. Other areas of 
investigation produced essentially null results. —-Awt/i. 
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Keseman, Charles E. "A Comparison of the Effect of 
Three Evaluation Approaches Upon Student Achievement 
in College Level Drafting" (Doctoral thesis). University 
of Missouri, Columbia, Miss*, 1967. 
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EVALUATION 



INNOVATION An Algorithm for the Design of Performance Criteria 

PROBLEM The present systems analysis of training manual does not 

provide much guidance as regards the strategies used m 
selecting appropriate work performance measurement 
strategies. Many of the program designers will be first- 
line supervisor/trainers, who are job-qualified, but who 
are not training experts. Procedures are needed to help 
them select the best strategy. 

SOLUTION A simple algorithm (to be incorporated into the Air 

Force Instructional System Development manual) that would 
help those responsible for the design of training pro- 
grams to select appropriate performance-measurement 
strategies . 

ADVANTAGES Decision making by those responsible for the systems 

analysis of Air Force training programs would be easier. 

RESOURCE The previously developed algorithm should be expanded in 

REQUIREMENTS scope and incorporated into a new chapter of the manual 
when it is routinely revised. 

SELECTED Angell, David, Shearer, James W. , and Berliner, David C. 

REFERENCES Study of Training Perfomanae Evaluation Techniques , 

prepared by American Institutes for Research, Palo Alto, 
Calif., for the U.S. Naval Training Device Center, Port 
Washington, New York, 16 October 1964. 

Finch, Curtis R. and Impellitteri, Joseph T. "The 
Development of Valid Work Performance Measures," Journal 
of Industrial Teacher Education^ Vol. 9, No. 1, Fall 
1971, pp. 36-49. 

McKnight, A. James. "Work Unit Stock: Development of 
Training Management Procedures for Heterogeneous Ability 
Groups ," in Use of Job and Task Analysis in Training^ 
the George Washington University, Human Resources Re- 
search Office, HumRRO Professional Paper 1-69, Januar>' 
1969, pp. 4-11. (Presentations at Headquarters, U.S. 
Continental Army Command, Fort Monroe, Va. , October 
1968.) (AD 688 810) 

This paper records the four presentations on the "L'je of 
Job and Task Analysis in Training" made by members of 
the HumRRO staff at a briefing sponsored by the Oftice 
of the Deputy Chief of Staff for Individual Training at 
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Headquarters, U.S. Continental Array Command in October 
1968. The presentations specifically describe job :;nd 
task analysis and its role in curriculum engineering. 
The briefing was designated the first of a series of 
briefings on training research and development programs 
of the U.S. Army Behavioral Science Research Laboratory, 
the Center for Research in Social Systems, and HumRRO.-- 
Auth. ^ 

Schriber, Peter E. (Pennsylvania State University.) 
An Empivioal Compariaon of Criterion-Refereneed Data 
Collected by Mastery Testing Versus Repeated Item- 
Examinee Sampling, 1973. (Paper presented at the annual 
meeting of the American Educational Research Associa- 
tion, New Orleans, 1973, as part of the Symposium on 
"Criterion-Reference Evaluation Using Time-Series De- 
signs and Item-Examinee Sampling.") 

Siegel, Arthur I., Schultz, D.G., and Federman, Philip. 
Post'Training Performance Criterion Development and 
Application: A Matrix Method for the Evaluation of 
Training, prepared for the Personnel and Training 
Branch, Office of Naval Research, by Applied Psycholog- 
ical Services, Wayne, Pa., January 1961. 
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EVALUATION 



INNOVATION Management by Objectives Programs for E6s and Above 

PROBLEM Most E6s and above really need management training 

rather than technical training. Yet, the current train- 
ing regulations require that records be maintained on 
jobs, or Specialty Training Standards. The requirement 
is often ignored, since such records are not relevant 
to the real training needs of the trainees. Meanwhile, 
important training objectives that should be considered 
are not discussed. 

SOLUTION Drop the requirement for keeping records in terms of STS 

categories for E6s and above; replace it with a manage- 
ment-by-objectives program patterned after those that 
are so successful in industry. These programs operate 
by establishing expected performance objectives (with 
clear-cut standards of performance) during joint plan- 
ning conferences attended by a supervisor and his sub- 
ordinate. The progress of the subordinate during a 
specified period of time is then reviewed in terms of 
the agreed upon standards of performance, and new objec- 
tives are established for the next reporting period. 

More meaningful training plans and training records for 
E6s and above, with consequent increases in the quality 
of training. 

Research is needed to design and field test a manage- 
ment by-objectives approach for noncommissioned officers. 
Time requirements during planning conferences should 
average approximately one-half hour each. This is more 
time than is taken today, because many people do not 
take the present requirements seriously. 

SELECTED Odiome, George S. (University of Utah). Training by 

REFERENCES Gbjeatives: An Economic Approach to Management Training. 
The Macmillan Company, New York, 1970, 354 pp. 

Pajer, Robert G. Systems Approach to Results-Oriented 
Performance Evaluation,** Personnel Administration and 
Public Personnel Review^ Vol. 1, No. 3, November- 
December 1972, pp. 42-47. 



ADVANTAGES 



RESOURCE 
PEQUIREMENTS 



ERLC 



132 



EVALUATION 



INNOVATION Results-Oriented Procedures for Headquarters Evaluations 
of Training Programs 

PROBLEM Inspector General and MAJCOM inspection teams tend to 

put too much emphasis on record-keeping procedures when 
they evaluate OJT programs. This has a number of unin- 
tended consequences (e.g., falsification of records), 
and generates resentment on the part of supervisors who 
have outstanding training programs but who have not had 
time to maintain their records properly. Some of the 
record-keeping practices, moreover, are rather trivial 
in nature (e.g., a supervisor c^uld be written up for 
making an entry in pencil ratht ban in ink, or vice 
versa) . 

SOLUTION Design new results-oriented procedures for evaluating 

OJT programs of the type that have a demonstrable re- 
lationship to organizational unit objectives. 

ADVANTAGES The present overemphasis upon the details of record- 
keeping procedures would decrease; attitudes towards 
the OJT program would improve; and the real quality 
(rather than the paper-work appearance) of OJT programs 
would be evaluated by inspection teams, resulting in 
improvements in program quality as opposed to improve- 
ments in paper work. 

A fairly large-scale research study would be needed to 
identify new inspection procedures that could be used by 

Force insp^^ction teams. A minimum of 20 squadrons 
would be needed for a definitive study, and much pre- 
liminary work would be needed first. Manpower require- 
ments are projected at three man-years. 

SELECTED Warren, M.W. ^'Analysis of Training Needs,** Chapter 4 

REFERENCES in Warren, M.W., Training for Results: A Systems Ap-- 
proaah to the Development of Human Resources in Indus-- 
try^ Addison-Wesley , Reading, Mass., 1969, pp. 47-66. 

Warren, M.W. ^'Administrative Skills Training,** in 
Warren, M.W. , Training for Results: A Oy^iems Approach 
to the Development of Human Resources in Industry y 
Addison-Wesley, Reading, Mass., 1969, pp. 149-150. 
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EVALUATION 



INNOVATION 



PROBLEM 



SOLUTION 



ADVANTAGES 



RESOURCE 
REQUIREMENTS 



COMMENT 



More Use of Criterion- Referenced Measurement in the 
Design of End-of -Course Examinations and Specialty 
Knowledge Tests 

End-of- course examinations and Specialty Knowledge Tests 
tend to be norm- referenced rather than criterion- refer- 
enced; i.e., Air Force personnel often pass examinations 
because they get higher scores than other personnel, 
rather than because they can meet the requirements of 
the position. This is especially true of SKTs, m which 
those with the highest scores tend to get promoted as 
part of the \ APS program. The difficulty with norm- 
referenced testing is that it tends to put too much em- 
phasis on verbal skills and memory— these, moreover, are 
more likely to be valued by and characteristic of middle- 
class Americans than they are of lower-class Americans. 
Thus, it can be argued that the tests are discriminatory. 
Recent Supreme Court rulings have held that many tests 
of this type are discriminatory, which gives the whole 
matter some priority. 

Conduct research to identify criteria for deciding when 
verbal memory tests should be permitted and when they 
should not. Identify which tests are inadequate by 
these criteria, then design criterion- referenced tests 
for those cases in which the verbal memory tests are 
considered inadequate or unfair. 

Improved fairness and demonstrable job relevance of 
tests . 

Approximately 200 man-years of effort would be required 
to design performance standards for all Air Force spe- 
cialties and convert them into criterion- referenced 
tests. Assuming that only 25 percent really need it, 
the total costs are estimated at 50 maii- years. 

The Air Force is embarked upon a large-scale effort 
called "systems analysis of training" that could con- 
ceivably result in a large number of criterion-refer- 
enced tests. But, if Air Force budgets are cut in ac- 
cordance with present plans, it will probably be decided, 
in most cases, that criterion-referenced tests are too 
expensive. 
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SELECTED Cox, Richard C. ''Evaluative Aspects of Criterion- 
REFERENCES Referenced Measures/' in Popham, W. J. (edO> Criterion-^ 
Refevenoed Measurement y An Introduction ^ Educational 
Technology Publications, Englewood Cliffs, N,J,, 1971, 
pp. 67-75. 

Davis, Frederick B. Criterion^-Referenced Measurement, 
1971 AERA Conference Summaries, TM Reports, No. 12, 
ERIC Clearinghouse on Tests, Measurement, and Evaluation. 
Educational Testing Service, Princeton, N.J., March 
1972. 

Garvin, Alfred D. ''The Applicability of Criterion- 
Referenced Measurement by Content Area and Level," in 
Popham, W, James (ed.), Criterion-Referertaed Measure-- 
mentj An Introduotiony Educational Technology Publica- 
tions, Englewood Cliffs, N.J., 1971, pp. 55-63. 

Glaser, Robert A. "A Criterion-Referenced Test," in 
Popham, W. James (ed.), Criterion-Referenaed Measure- 
ment j An Introduction^ Educational Technology Publica- 
tions, Englewood Cliffs, N.J., 1971, pp. 41-51. 

Glaser, Robert. "Instructional Technology and the 
Measurement of Learning Outcomes: Some Questions," in 
Popham, W. James (ed.), Criterion-Referenced Measure- 
mentj An Introduction^ Educational Technology Publica- 
tions, Englewood Cliffs, N.J., 1971, pp. 5-14. {See 
also same title, American Psychologist^ Vol. 18, No. 8, 
August 1963, pp. 519-521.) 

Glaser, Robert and Klaus, David J. "Proficiency Measure- 
ment: Assessing Human Performance," in Gagn^, R.M. 
(ed.). Psychological Principles in System Development^ 
Holt, Rinehart and Winston, New York, 1962, pp. 419-474. 

Popham, W. James (ed.), Criterion-Referenced Measure- 
ment^ An Introduction^ Educational Technology Publica- 
tions, Englewood Cliffs, N.J., 1971. 

Popham, W. James and Husek, T.R. "Implications of 
Criterion-Rererenced Measurement," in Popham, W. James 
(ed.), Criterion-Referenced Measurement y An Introduc- 
tiony Educational Technology Publications, Englewood 
Cliffs, N.J., 1971, pp. 17-37. {See also same title. 
Journal of Educational Measui^ementy Vol. 6, No. 1, 
Spring 1969, pp. 1-9.) 




Shriver, E.L. (Principal Investigator) and Hitchcock, 
CD. (Associate Investigator). "Field T"TOut and Eval- 
uation of Job-Task Perfoirraance Tests," in Work Unit 
SuimarieSy Defense Documentation Center, Defense Supply 
Agency, Alexandria, Va. , DDC Report No. CT9150, 22 August 
1972, p. 44. (Performing Organization: U. R. S. Corp., 
San Mateo, Calif.; Contract No. F33615"70-C'1695.) 

Underwood, B.J. (Principal Investigator). "Personnel 
Technology: Defining the Conditions l^hich Control How 
Well Text Material Is Learned and How Long It Is Re- 
membered," in Work Unit Sumnaries^ Defense Documenta- 
tion Center, Defense Supply Agency, Alexandria, Va., 
DDC Report No CT9150, 22 August 1972, p. 121. (Per- 
forming Organization: Northwestern University 
Psychology Department, Evanston, 111.; Contract No. 
N00014-67-A-0356-0010. ) 
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INCENTIVES-MOTIVATION 



INNOVATION Written Training Contracts Between Instructors and 
Trainees 

PROBLEM Trainees have little control over their training, con- 

sequently they do not have as personal a conunitment to 
achieve training objectives as they might have if they 
were more involved in the goal -setting process. 

SOLUTION Written training contracts between instructors and 

trainees, in which the trainee requests training in 
certain tasks (or CDC volumes), and commits himself 
to a certain time span for learning those tasks (or 
CDC volumes) . 

ADVANTAGES Improved motivation on the part of trainees; more in- 
volvement of trainees in the training process; improved 
selection of tasks to meet the needs of trainees. 

RESOURCE This idea needs to be elaborated upon and field-tested 

REQUIREMENTS before it is implemented. Resource requirements for 
the development and evaluation work are estimated at 
two man-years. Implementation costs are estimated at 
one-half hour per trainee. 

SELECTED Bialek, Hilton and McNeil. Michael. Preliminary Study 

REFERENCES of Motivation and Incentives in Basic Combat Training, 

Technical Report 68-6, The George Washington University, 
Human Resources Research Office, HumRRO Division No. 3, 
Presidio of Monterey, Calif., May 1968. 

Teel, Dean A. Comparison of Methods Utilizing the 
Contract Approach in Teaching Beginning Electricity- 
Electronics Fundamentals to College Students" (Doctoral 
thesis), Texas ASM University, College Station, Texas, 
1967. 
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INSTRUCTIONAL TECHNIQUES 



INNOVATION Performance-Based Instruction Techniques 

PROBLEM The Air Force currently handles subject matter training 

separately from proficiency training, as a basic concept 
of the dual-channel OJT concept. In fact, there are 
many advantages in integrating the two types of train- 
ing so that the trainee can acquire information in job- 
relevant situations. The situation is complicated by 
supervisors being expected to design proficiency tests 
of their own. Consequently, there is no consistent 
standard, except for the STS requirements. There is 
considerable duplication of effort by supervisors at 
different locations. 

SOLUTION Design performance tests with specified behavioral per- 

formance standards and use these tests for proficiency 
training at the outset rather than waiting for some 
preliminary period of instruction. 

ADVANTAGES Students learn in a job-related situation; each student 
is required to reach a standard of performance in each 
skill; emphasis is on active skill performance, not 
passive absorption of information; student motivation 
is consequently better than usual. 

Behavioral objectives are expected to result from the 
Air Force systems analysis of training movement that 
is currently in progress. The conversion of these be- 
havioral objectives to performance-based instruction 
techniques would require about $10,000 per specialty. 
This cost and that of printing and disseminating the 
information would be less expensive. There is reason 
to believe that there is a considerable duplication of 
effort in the present approach. 

SELECTED Taylor, John E. , Michaels, Eugene R. , and Brennan, Mark F. 

REFERENCES The Concepts of Performnae-OHented Ineti-uction Used ^n 
Deve''cping the Experimental Volunteer Amy Training Pro- 
gram, HumRRO Technical Report 72-7, Human Resources Re- 
search Organization, HumRRO Division No. 3, Presidio 
of Monterey, Calif., March 1972. 
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This report describes the planning and implementing of 
the Experimental Volunteer Army Training Program (EVATP) 
at Fort Ord early in 1971. This was the Army's first 
effort to effect major training innovations in the con- 
version toward an all-volunteer Army. By the fall of 
1971, this program was being used as a model for imple- 
menting the EVATP at other Army Training Centers. In 
developing the EVATP system, six established learning 
principles were applied to Basic Combat Training and 
Advanced Individual Training to modify the conventional 
training system. Course objectives and performance 
tests used were developed jointly by Fort Ord and HuraRRO. 
In a comparison with a conventionally trained group, in- 
dependently conducted by the Infantry School at Fort 
Benning, EVATP graduates performed significantly better 
on five out of seven BCT subjects, and seven out of nine 
Air subjects. In general, these gains were shown by men 
at all levels of aptitude. --ylutft. 

COf-IMENT The notion of modularizing training can also be employed 

to make objectives more job-relevant and performance- 
oriented. The Army's "duty module*' concept is an example 
of what can be done with this approach, 

ADDITIONAL Miller, Robert S- C\merican Institutes for Research, 
REFERENCES Washington, D.C.). A Taxonomio Base for Future Manage- 
ment Infomiation and Decision Systems: Theoretical 
Background to the Design of Duty Modules ^ prepared for 
U.S. Army Behavior and Systems Research Laboratory, 
Office of Chief of Research and Development, Department 
of the Army (Technical Research Note, July 1971, un- 
published) . 

Stephenson, Robert (American Institutes for Research, 
Washington, D.C.). A Taxonomic Base for Future Manage- 
ment Information and Decision Systems: A Common Language 
for Resources and Requirement Planning, Technical Research 
Note 244, prepared for U.S. Army Behavior and Systems 
Research Laboratory, Office of Chief of Research and Develop- 
ment, Department of the Army, October 1972, (AD 757 794) 





INSTRUCTIONAL TECHNIQUES 



INNOVATION 
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SOLUTION 



ADVANTAGES 



RESOURCE 
REQUIREMENTS 



COMMENT 



Peer Instruction 

Individualized rates of instruction are preferable, be- 
cause slow learners can work at a comfortable pace, and 
fast learners are not forced to wait for the slower 
ones, thereby wasting their time. It is not practical 
to have a separate instructor for each aptitude level 
of personnel; but lower-aptitude personnel seem to need 
more help and assistance than higher-aptitude personnel. 

iup fellow trainees as instructors (peer instructors) 
when performance criteria are well defined. Unusually 
bright students can be used in most cases, but it seems 
to be desirable to use a low-aptitude trainee who has 
completed a block of instruction, especially when the 
problem seems to be the attitude of another low-aptitude 
trainee who has not completed training. 

Individualized instruction is made practical under many 
circumstances that would not permit it otherwise; lower- 
aptitude personnel get more help. 

The effectiveness of this approach depends upon circum- 
stances, so each experiment with it is just that- -an 
experiment. Initial development costs could be high 



SELECTED 
REFERENCES 



exper 
(perh . 

aiT capacity and 



(perhaps two man-year.s), but the potential for increased 
• individualization is worth evaluating. 



This approach should only be used when performance cri- 
teria are well defined. The peer instructors cannot be 
permitted to certify completion of training, and those 
who do certify completion must have some opportunity to 
observe the trainee. 

Human Resources Research Organization. "Peer Instruc- 
tion," Training in Business and Industry, Vol. 9, 
No. 3, March 1972, pp. 38-42. 

Weingarten. Kenneth et al. The Apstrat Instruotional 
Model, HumRRO Professional Paper 6-71, Human Resouices 
Research Organization, Alexandria, Va. , May 1971. 
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Weingarten, Kenneth; Hungerland, Jacklyn; Brennan, Mark; 
and Allred, Brent, The Development of a Low^-Cost Per-- 
formanae^Oviented Training Model, HumRRO Professional 
Paper 32-70, Human Resources Research Organization, 
Alexandria, Va. , December 1970, (Papei i resented at the 
American Psychological Association Convention, Miami 
Beach, Fla., September 1970.) 

This paper describes a training model featuring peer 
instruction in a functional job-simulated context, at 
well as the objectives and practical constraints that 
led to its development.— i4Mt7z. 

Air Force Human Resources Laboratory. ''Research to 
Identify and Evaluate Peer Instruction Applications for 
Air Force Technical Training,'' in Work Unit Summaries, 
Defense Documentation Center, Alexandria, Va., DDC Re- 
port No. Z00966, 5 March 1974, p. 8. (Performing Organi- 
zation: Air Force Human Resources Laboratory, Lowry 
AFB, Colo.) ^ 
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INSTRUCTIONAL TECHNIQUES 



RESOURCE 
REQUIREMENTS 



INNOVATION Utilization of Talent Ratings to Identify Problem 
Specialties 

PROBLEM The Airman's interest in his job and the felt utiliza- 

tion of talent varies markedly from specialty to spe- 
cialty, even though each specialty has similar promo- 
tion prospects, and the various specialties are rela- 
tively the same. Trainees who are assigned to the lower 
interest specialties (which must be assigned to someone) 
tend to get bitter and leave the Air Force after their 
first enlistment. If they do not leave, they will re- 
quest reassignment and retraining into another specialty 
as soon as they have met their minimum time requirements. 
Either action is expensive for the Air Force. 

SOLUTION Scales measuring job interest and utilization of talent 

have been placed in all Air Force Occupational Surveys 
administered since 1966. These data can be analyzed and 
used to identify problem specialties that require re- 
medial action of some kind. 

Periodic statistical analyses of job interest and felt 
utilization of talent ratings by various groups of 
trainees must be followed up by special studies that 
would identify the exact problem. Some studies may be 
a matter of a few man-weeks by a committee, but others 
could conceivably involve large-scale field surveys with 
a representative sample of airmen in the specialty. 

ADVANTAGES Improved retention and job satisfaction of Air Force 
personnel in the specialties affected. 

COMMENT There is often a separation of personnel research and 

personnel operations groups that acts as a hindrance to 
the implementation of research findings. This should 
be considered when study teams are established. 

SELECTED Christal, Raymond E. Analysis of Racial Differences in 
REFERENCES Terms of Work Assignments, Job Interest, and Felt 
Utilization of Talents and Training, AFHRL-TR-72-1 , 
Personnel Research Division, Air Force Human Resources 
Laboratory, Lackland AFB, Texas, January 1972.AD-741 758. 

This study reports data analyses for first-term airmen 
in 11 career ladders to determine whether there are 
differences in work assignments and job attitudes of 
Blacks and Non-Blacks. The general approach involved 
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application of the multiple linear regression model to 
determine the relationships between race and selected 
criteria, holding constant such variables as aptitude, 
time in military service, technical school graduation 
status, and time on the job. No racial differences were 
observed in the number of tasks being performed or in 
the average difficulty of tasks performed per unit time. 
However, when these two factors were weighted into an 
overall job difficulty composite, it was found that the 
Blacks were performing slightly less difficult jobs in 
two of the career ladders; 60SX0 Air Passenger/Air Cargo 
and 702X0 Administrative. Significant differences in 
job interest and felt utilization were found in two lad- 
ders, 291X0 Communications Center and 702X0 Administra- 
tive; in each instance, these differences were in the 
direction of Blacks finding their jobs more interesting 
and feeling a greater utilization of their talents and 
training than Non- Blacks. Only a small proportion of 
the job assignment variance could be accounted for by 
all variables in the system, l^he unique contribution 
of race was significant in two ladders, but in each in- 
stance this contribution W2*i> less than one percent. 
There appear to be no pra::tical differences in the types 
0/ assignments given to Blacks and Non-Blacks in the 11 
ladders investigated. Blacks in the 291X0 and 702X0 
areas reported higher job interests and a higher feeling 
of utilization. Again, these differences were signifi- 
cant, but were relatively small,--i4wt/2. 

Gould, R. Bruce. Reported Job Interest and Perceived 
Utilization of Talents and Training by Hrmen in 97 
Career Ladders, AFHRL-TR-72-7. Personnel Research Divi- 
sion, Air Force Human Resources Laboratory, Lackland 
AFB, Texas, January 1972. AD-745 099. 

The purpose of this study was to investigate the extent 
of differences in reported job satisfaction of over 
100,000 airmen in 97 career ladders. The differences 
between career ladders and between individuals within 
career ladders were evaluated. Two seven-point scales 
measuring incumbents' job interest and feelings of how 
well their jobs make use of their talents and training 
have been included in inventories administered under 
the USAF Occupational Survey Program. Analyses of the 
responses indicated that while most airmen found their 
jobs interesting and felt well utilized, there were some 
extreme differences between career ladders and among 
individuals within ladders. Extensive ladder by ladder 
studies are warranted to identify factors relating to 
differences in job satis faction. 
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SELECTED 
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Periodic Surveys of Trainees 

Trainees have a great deal of information about the 
quality of training when they are undergoing OJT, but 
the information is not collected in a systematic manner 
or used to make decisions. Part of the problem is that 
the official communication channels are too formal and 
too serious for a trainee to feel comfortable using 
them. 

Conduct periodic surveys and anonymous interviews with 
trainees in which they are asked to rate the quality of 
their training while in OJT; identify problem areas; 
suggest improvements. 

Anonymous interviews will permit trainees to ''blow off 
steam/* as well as communicate information to people 
who are in a position to solve problems. Anonymous 
questionnaires have a similar effect. Both types of 
survey are also proof that the Air Force is concerned 
abc-t the welfare and job satisfaction of its trainees. 
More importantly, however, information would be collected 
that would permit improvements in the Air Force OJT pro- 
gram. 

Only a small sample of trainees need be interviewed. If 
the interviews are conducted by CBPO OJT personnel, it 
is estimated that 100 interviews per year would be suf- 
ficient at each large base. The CBPO OJT staff, in 
turn, could be asked to prepare an anonymous report, 
once every six months, in which potential problem areas 
would be identified and recommendations would be made 
for modifying the base OJT effort. The surveys should 
be conducted on an Air Force-wide basis, and should in- 
volve a sample of at least 100 trainees in every MAJCOM. 

Gessner, Peter K. ^'Evaluation of Instruction," Science^ 
Vol. 180, No. 408C, 11 May 1973, pp. 566-670-. 

It appears quite clear that student ratings of instruc- 
tion and class performance on national normative examina- 
tions are postively related: the higher the student 
ratings of the instruction they receive, the higher the 
class score relative to a n^^tionwide norm. On the other 
hand, no significant relationship exists between student 
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ratings and class performance on institutional 
examinations. This suggests that both student 
ratings and class performance on national normative 
examinations are valid measures of teaching 
effectiveness . — Auth. 
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INNOVATION Use of More Illustrations and More Simple Language in 
Technical Manuals and CDCs 

PROBLEM Many technical manuals have relatively few illustrations 

and are written at higher reading levels than most Airmen 
in a specialty can readily comprehend. 

SOLUTION U<^e more illf -trations and more simple language in tech- 

nical orders. C . and other job-related publications. 

ADVANTAGES Improved understanding and retention on the part of 
trainees, especially those with low aptitude. 

RESwRCE Editing materials for readability and more simple language 

REQUIREMENTS could cost approximately $5,000 per text . Additional il; 
KtguiKhntmi lustration costs would vary from publication to publication. 

COMMENT Revising CDCs will not help if there is no allowance for 

the technical manuals to be written at a lower reading 
level It seems logical to begin with the technical 
manuals required for job performance. If they cannot 
be improved. :;he Air Force should increase the abi.ity 
standards for the men assigned. 

SELECTED Sellman. Wayne S. Effectiveness of J ferment cd Training 

REFERENCES Materials for Low Ability Avrmen, AFHRL-TR-70-16 . Fecn- 
nical Training Division. Air Force Human Resources Lab- 
oratory. Lowry AFB. Colo.. June 1970.AD-717 712. 

Tlie studv was designed to determine if modifying Career 
Development Course (CDC) format through the simplifica- 
tion of the written materials, the inclusion of more 
illustrations, and the addition of audio supplementation 
could improve the CDC as a training device designed to 
teach basic job information, especially to airmen pos- 
sessing minimum verbal skills. High-, middle-, and low- 
aptitude personnel studied three versions of the CDL tor 
the 57130. fire protection, career ladder. In brief 
the versions included a conventional CDC. a less verbal 
CDC with more pictorial m&terials. and a less verbal CDL 
with more pictorial materials accompanied by a tape re- 
cording of information complementary to that contained 
in the written text of the CDC. Data were collected on 
leamin performance, reading speeds, and attitudes to- 
ward the CDC. The analyses revealed that tne modified 
CDC with the audio supplementation produced significantly 
increased learning scores. High- and middle-aptitude 
groups consistently outperformed the low-aptitude groups 
across all CDCs . --Aut^z. 
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INNOVATION Consider the Use of Job Performance Aids as an Integral 
Part of the Process by Which Training Programs Are 
Designed. 

PROBLEM Training programs and job-performance aids tend to be 

designed by different people and with relatively little 
coordination. As a result, some training design prob- 
lems that could be solved through the use of job per- 
formance aids are either otherwise resolved or not re- 
solved at all. 

SOLUTION Require that those responsible for the design of train- 

ing programs be trained in the design of job-perfonrance 
aids, and be kept informed regarding the plans of other 
Air Force agencies for job-performance aids so that they 
might be able to use them effectively. 

ADVANTAGES Simplification of training requirements where job-train- 
ing aids can help, thus resulting in a savings in train- 
ing time and possible increases in efficiency. 

RESOURCE An additional three days would be needed in the training 

REQUIREMENTS programs currently taken by technical writers/instructors. 

Some TDY would also be needed for conferences between 
job-performance experts and training-program designers. 

SELECTED Chalupsky, Albert B. and Kopf, Thomas J. Job Per for- 
REFERENCES manee Aids and Their Impact on Manpower Utilization, 
WDL-TR-3276, prepared for Office Manpower Policy, 
Evaluation and Research, U.S. Department of Labor, by 
Philco-Ford Corporation, WDL Division, Palo Alto, Calif., 
May 1967. 

Chenzoff, Andrew P., Mallory, William J,, and Joyce, 
Reid P. (Applied Science Associates, Inc.). Guidanae 
and Specification for the Preparation of Fully Proce-- 
duralized Job Aids for Organizational and Inteimediate 
Maintenance of Electronic Subay stems. Technical Report 
AFHRL-TR-71-.23, Air Force Human Resources Laboratory, 
Brooks AFB, Texas, June 1971. AD-731 144. 

This report supplies a model for specification for the 
preparation of fully proceduralized job aids for organi- 
zational and intermediate maintenance of electronic sub- 
systems, and offers guidance in the preparation of such 
aids. The aids to be developea from these specifications 
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are for flight-line or field-shop maintenance of any 
electronic subsystem, and support the performance of 
the following maintenance functions as needed: check- 
out, aligranent, repair, adjustment, calibration, mal- 
function localization, malfunction isolation, and the 
removal and replacement of malfunctioning equipment 
items. — Auth, 

Foley, John P., Jr. Job Performance Aids Research^ 
Summary^ and Recommendations^ Air Force Human Resources 
Laboratory, Wright-Patterson AFB, Ohio, April 1969. 
(AD 697 054) 

Gebhard, Richrrd M. development of a Training Program 
and Job Aids jor Maintenance of Electronic Communica- 
tion Equipment, Human Resources Research Organization, 
HumRRO Division No. 1, Alexandria, Va. , Technical Re- 
port 70-19, December 1970. 

This study was designed to develop and evaluate methods 
for producing a combination of training and job aids 
(manuals) for maintenance of electronic communication 
equipment that would require less training time than 
the standard course and manual s.--i4wt/z. 

Hoehn, Arthur J. and Lumsdaine, Arthur A. design and 
Use of Job Aids for Cormunicating Technical Information^ 
Technical Report AFPTRC-TR-58-7 , Maintenance Laboratory, 
Air Force Personnel and Training Research Center, Air 
Research and Development Command, Lowry AFB, Colo., 
January 1958. (AD 152 109) 

Serendipity, Inc. Project PIMO Final Repoyt: PIMQ 
Final Report Summary, May 1969. (AD 852 101) 

Singer, H.J. and MacDonald, I.D. Is Apprenticeship 
Outdated?, Institute of Personnel Management, Oxford 
Circus , London, 1970. 
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INNOVATION Provide Task Difficulty Ratings to Those Responsible 
for OJT 

PROBLEM Every supervisor in the Air Force is expected to act as 

a trainer for his subordinates, even when he has not 
had previous experience in the tasks for which he is 
providing training. As a result, so?\e trainees are as- 
signed tasks thar are too difficult or too easy for 
their state of progress, 

SOLUTION A procedure for collecting task difficulty ratings has 

bee.n designed by the Air Force Personnel Laboratory, 
These procedures could be applied to STS categories 
when STSs are reviewed and revised a routine basis. 
The information could then be systematically incorpo- 
rated into the STSs, and used to guide inexperienced 
supervisor/trainers , 

RESOURCE Approximately cwo man-days of committee time for each 

REQUIREMENTS specialty would be required when the STSs are routinely 
reviewed by a panel of NCOs, No printing costs would 
be involved if the changes were restricted to new or 
revised STSs, 

ADVANTAGES More appropriate assignments for trainees at various 
levels of skill and aptitude, 

COMMENT This sort of information is not needed by experienced 

trainers. There may also be restrictive conditions 
that make some tasks more difficult than the por.cl of 
experts would normally have expected them to be, 

SELECTED Lecznar, William B, Three Methods for Estimating 

REFERENCES Difficulty of Job Tasks^ AFHRL-TR-71-30, Personnel 
Division, Air Force Human Resources Laboratory, 
Lackland AFB, Texas, July 1971, AD-730 594, 

The numbers and ability levels of persons who might en- 
list in the Air Force can be projected only within 
gross boundaries. It is necessary^ however, to make 
optimal use of whatever manpower is available in the 
enlisted force. An important tool for this purpose is 
accurate information about the difficulty of tasks that 
make up Air Force jobs and the abilities of airmen as- 
signed to perform those tasks. This study explores 
two methoGT, of obtaining estimates of task difficulty 
and evaluates the comparative merits of the methods, 
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Using the Medical Materiel Career Ladder (915X0) job 
inventory as the vehicle for the study, senior noncom- 
missioned officers judged each of the task statements 
on a relative scale of difficulty, A second group of 
NCOs rank ordered the tasks according to their esti- 
mates of difficulty. The tasks were then arranged ac- 
cording to the average grade level of airmen who stated 
they performed each task, A mathematical estimate was 
made of the agreement on task difficulty within the 
group of raters, within the group of rankers, and be- 
tween the difficulty orders established by ratings, 
rankings, and grade. Considerable agreement was found 
among both the raters and the rankers as to the dif- 
ficulty of tasks. Tne relationship between rated dif- 
ficulty and ranked difficulty was substantial and was 
higher than either of these with the order of tasks 
based on average grade level. The study did not, how- 
ever, treat the question of the accuracy of any of the 
judgments, A number of studies in Air Force occupa- 
tional research depend upon judgments of workers in 
various career ladders. It seems clear that the method 
of rating a large number of items on a relative scale 
will provide data from pooled judgments which can be 
used with confidence and which ran be ubtained with 
less time expenditure than that required for rank 
ordering. Because of the lower relationships obtained, 
use of average grade as an index of task difficulty 
does not appear warranted for general application to 
evaluation of task attributes, --y4ut?i. 

Mead, Donald F, Continuation Study on Development of 
a Method for Evaluating Job Difficulty^ AFHRL-TR-70-43, 
Personnel Division, Air Force Human Resources Labora- 
tory, Lackland AFB, Texas, November 1970, AD-720 254, 

There has been no tool available by which Air Force 
personnel inaiiagers could evaluate jobs in terms of 
theii* ordering on specified dimensions for use in 
various policy and decision making situations. The 
problem attacked in this <^tudy was to determine whether 
supervisors' judgments of job difficulty (as one spe- 
cific dimension) could be predicted with sufficient 
accuracy from certain information readily accessible. 
With the Medical Career Ladder as the area for study, 
senior noncommissioned officers were asked to rank 
order actual jobs performed by ainnen in the ladder; 
other NCOs were asked to provide ratings of difficulty 
on a relative scale for each of the tasks listed in 
the job inventory previously developed for the ladder. 
Through application of statistical techniques, the 
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rankings » ratings, and other closely related data were 
treated to yield a prediction system. The stability of 
the system was then tested. From 21 variables and 17 
multiple regression problems, an optimally weighted 
equation was derived which apparently captured the 
policy used by the judges in rank ordering the jobs on 
estimated difficulty. The equation contained three 
predictors: Number of Tasks Performed, Average Task 
Difficulty per Unit Time Spent, and Number of Tasks Per- 
formed, Squared; an i? of ,95 and an of .90 were ob- 
tained between the predicted job difficulty values and 
the judged job difficulty ranks. Cross-application of 
the equation to test for stability yielded an i? of .94 
in both samples. Further analysis of the predicted 
values showed that 90 percent were within 2.75 points 
of the ranked values (on a scale of 1 through 25). The 
study indicates that the job difficulty equation de- 
rived from information easily collected by means of job 
inventories will predict ranked job difficulty with a 
high degree of accuracy, at least in the Medical Mate- 
riel Career Ladder, If further research in other 
career ladders verified the same constituent elements 
with similar weights, it will be possible to have an 
index of difficulty for each job in all airman career 
fields. The implication is that the Air Force person- 
nel system would be given material assistance toward 
establishing grade requirements, in considering job 
aptitude minimums, for comparing work assigned to var- 
ious categories of personnel, and other related per- 
sonnel actions. — Autk. 

Mead, Donald, F. Development of an Equation for Eval- 
uating Job Difficulty, AFHRL-TR- 70-42, Personnel Divi- 
sion, Air Force Human Resources Laboratory, Lackland 
AFB, Texas, November 1970. AD-720 253. 

In a prior study an equation was developed which ac- 
curately predicted supervisors* judgments of the dif- 
ficulty of airman jobs in the Medical Materiel Career 
Ladder. The utility and effectiveness of this tool for 
Air Force personnel managers would be considerably en- 
hanced if the equation could be generalized to all Air 
Force specialties. The present study was directed to 
the question of whether an equation developed on a 
second specialty. Vehicle Maintenance, had the same or 
similar characteristics as the earlier one. Rank 
orderings of a random set of job descriptions, based on 
job analysis of the Vehicle Maintenance Career Ladder, 
and difficulty ratings on a 7-point relative scale of 
each task in the Vehicle Maintenance job inventory were 



requested of separate groups of noncommissioned officers 
working in the specialty. By application of a policy- 
capturing technique, an equation was derived to predict 
the job difficulty ranks. The Vehicle Maintenance job 
difficulty equation predicted with a high degree of 
accuracy (i? = .93) supervisors* judgments of job diffi^ 
culty. The primary factors reflected in the supervisory 
judgment of job description, difficulty level of tasks 
performed, and time spent performing the tasks. The 
three basic variables which combined to capture the job 
difficulty evaluation policy were the same as those in 
the prior study. On cross-application, the equation 
showed no appreciable loss in effectiveness. Major 
findings replicated those in the Medical Materiel study. 
Apparently common factors exist across career ladders 
which enter into supervisors' judgments of job diffi- 
culty; within each ladder the supervisors' evaluation 
policy was captured in quantified form. There remains 
the unresolved question of whether an effective compos- 
ite standard weight equation can be developed for eval- 
uating jobs in all airman career 1 adders. -->lwt/i. 

Mead, Donald F. and Christal, Raymond E. Development 
of a Constant Standard Weight Equation fov Evaluating 
Job Diffioulty, AFHRL-TR- 70-44, Personnel Division, Air 
Force Human Resources Laboratory, Lackland AFB, Texas, 
November 1970. AD-720 255. 

Availability of a device which would provide valid 
measures of some specified important factor in Air 
Force jobs could be used to i^rove such features of 
the Air Force persornel system as assignment-reassign- 
ment, career planning, and force development. An 
initial study developed such a technique for evaluating 
the factor of job difficulty; a following study verified 
the procedure in a second career field by replicating 
the results of the initial research. The present ef- 
fort, in addition to checking the generalizability of 
the procedure to a third airman specialty, focused on 
derivation and test of a ^'universal** equation that 
could be given to personnel managers as a simple-to-use 
effective quantitative indicator of job difficulty. 
Senior noncommissioned officers in the Accounting and 
Finance Career Ladder provided their judgments on the 
rank order difficulty of 250 jobs which were described 
in terms of actual tasks performed by incumbents. A 
second group of senior NCOs each rated all the tasks 
listed in the job inventory for this ladder on a rela- 
tive scale of difficulty. Statistical techniques were 
applied to these data to simulate the judges' policy in 
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evaluating job difficulty, to test the efficacy of the 
derived prediction equation, and then to develop and 
test a set of standard weights to be assigned to each 
of the elements in the equation. 

The correlation of .95 between supervisors' ranking of 
job difficulty and predicted values based on the de- 
rived equation is highly satisfactory and replicates 
values-obtained in the two prior studies of different 
career fields. This study also confirms that the same 
three predictors found in the prior efforts combined 
to capture the evaluation policies of supervisors. On 
the main point of the study, a composite set of standard 
score weights derived from data in three ladders was 
demonstrated to provide highly acceptable validity 
(i? = .947Q for Medical Materiel, i? = .9247 for Vehicle 
Maintenance, and R = .9460 for Accounting and Finance). 
Differences between these values and validities yielded 
by optimal equations developed within the three ladders 
(.9486, .9269, and .9511, respectively) were nonsignif- 
icant. The indications are that the "universal" equa- 
tion can be used to obtain difficulty values for jobs 
across the total airman classification structure as 
quickly as occupational surveys are constructed and 
administered; only task difficulty values would have to 
be provided, and that aspect could be built into cur- 
rent job specialty surveys. The system is relatively 
free of external influences. Its most outstanding 
feature is the quantitative format which lends itself 
to existing automated data processing. The implica- 
tions are significant when one considers possible ap- 
plication of the syc*-Om to use in reassignment systems, 
in establishment of aptitude requirements, in comparison 
of work assigned to various categories of input (such 
as directed duty assignees, by-pass specialists, or 
technical school graduates), in investigation of inter- 
action between job difficulty, job satisfaction, and 
career intent, and in guidance of policy on force compo- 
sition. 



Miller, Robert B. "Task Description and Analysis," 
in R.M. Gagn^ (ed.). Psychological Princip7es in System 
Deoelopmnt, Holt, Rinehart and Winston, New York 
1962, pp. 187-228. 
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INNOVATION Decision Tables That Suggest Different Training Pro- 
cedures for Different Kinds of Tasks 

PROBLEM Many options exist, and it is difficult to decide which 

training procedures should be used under which circum- 
stances, 

SOLUTION Decision tables are to be based upon the following five 

categories of information: 

(1) The extent to which each of five defined "ongoing 
activities** is involved in the task; 

(2) The temporal, sequential, and casual relationships 
among these activities; 

(3) Characteristics of the detailed behavior that 
constitute the activities; 

(4) Contingencies that might affect task performance; 

(5) Disruptive conditions under which the task might 
have to be performed. 

The five ongoing Pwtivities include (a) Procedure 
Following, (b) Continuous Perceptual Motor Activity, 
(c) Monitoring, (d) Communicating, (e) Decision Making 
and Problem Solving. 

ADVANTAGES Expert decisions can be made by personnel who are not 
well read in the field. 

RESOURCE i:xi sting procedures can be incorporated into training 

REQUIREMENTS manuals; new procedures could also be designed, but 
would require research. 

SELECTED Folley, John D. , Jr. (Applied Science Associates, IncO- 

REFERENCES Development of an Improved Method of Task Analysis and 
Beginnings of a Theory of Training^ Technical Report 
NAVTRADHVCEN 1218-1, prepared for U.S. Naval Training 
Devices Center, Port Washington, New York, 22 June 1964. 

Haggard, Donald P. (U.S. Army Armor Ilumnn Research Unit, 
Fort Knox, Kentucky). The Feasibility of Developing a 
TaoP Classification Structure for OrdeHng Training 
Principles ani Training Content^ The Hcorge Washington 
University, Human Resources Research Office, Research 
Momornnduni, January 196.'5. 
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INNOVATION Revise STSs So That Certain Tasks Are Requiied 

PROBLEM Under the Air Force dual-channel system of OJT, the 

supervisor can, if he chooses, assign a single task 
to a trainee, and still get the trainee upgraded to 
the 5-skill level. The situation is complicated by 
the fact that some isolated Air Force locations do 
not have the equipment or the available supervisor/ 
trainer time in order to provide training in a reason- 
able variety of tasks. As a result, some of those up- 
graded to the 5-skill level are really not qualified, 
and they may not have received training in important 
safety procedures. 

SOLUTION Establish flexible minimum ti^aining goals for selected 

specialties and skill levels. This may take the form 
of listing a set of important tasks on the first page 
of the STS, and then requiring that some certain number 
(©•g., 3 out of 10) must be learned by any trainee up- 
graded to the 5-skill level. 

ADVANTAGES Trainees who transfer from one location to another will 
be more able to perform at their officially recognized 
skill level. 

RESOURCE Changes could be implemented when STSs are revised or 

REQUIREMENTS reviewed on a routine basis. 
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REFERENCES 



Locations that are not able to provide the type of train- 
ing tliat is needed would be assigned personnel who are 
already qualified. 

Elkin, Albert (U.S. Army Infantry Human Research Unit, 
Fort Benning, Ga.). "The Development of a List of Mini-^ 
mal Training Goals fov Basic Combat Training, HumRRO 
Technical Report 67, The George Washington University, 
Human Resources Research Office, December 1960. 

The Basic Combat Training Program (ATP 21-114, Nov 58) 
was analyzed in relation to each of 17 supporting Army 
Subject Schedules. Discrepancies between the ATP and 
its referenced subject schedules were noted and revi- 
sions suggested. On the basis of this analysis, a list 
of minimum training goals wdb devised for each subject 
presented in the report. These suggested training goals 
cover the minimum knowledge and skills needed by the 
individual basic combat trainee.- -4 wt/z. 
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Procedures for Deciding Which Training Techniques Should 
Be Used for Personnel at Different Aptitude Levels 

The best strategy to use in training someone seems to 
depend upon his aptitude level. High-aptitude personnel 
learn faster, need less guidance and repetition of in- 
struction--but the differences vary from one type of 
task to another. Research conducted as a result of 
Project 100,000 has identified many specific conditions 
under which additional guidance and repetition is needed, 
but decision tables based upon the characteristics of 
the task have not been designed. 

Design decision tables for deciding which training tech- 
niques should be used for personnel at different apti- 
tude levels, and incorporate them into training manuals 
for supervisor/trainers. 

More consideration for the special training needs of 
high- and low-aptitude personnel when hands-on training 
programs are designed. 

Research is needed to design and field test the use of 
the decision tables. 

Fox, W.L., Taylor, John E. , and Caylor, John S. Aptitude 
Level and the Acquisition of Skills and Knowledges in a 
Variety of Militaxy Training Tasks^ HuraRRO Technical Re- 
port 69-6, The George Washington University, Human Re- 
sources Research Office, HumRRO Division No. 3, Presidio 
of Monterey, Calif., May 1969. 

The Army has the problem of training men of widely dif- 
fering aptitude levels in a variety of military jobs. 
Recent Department of Defense decisions to lower mental 
standards for induction and enlistment to the statutory 
minimum AFQT score is resulting in a greater concentra- 
tion of lower aptitude trainees in the Army training 
program. Increasing the number of low-aptitude trainees 
will not only make the training job more difficult but 
may also result in marked loss in performance by the 
more apt as they become even more bored and restless 
than evidenced in the past. 
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Current technology of training provides little informa- 
tion useful to the Armed Forces for designing training 
programs to accommodate the entire spectrum of aptitude. 
Although research directed toward engineering of train- 
ing for those in lower mental Category IV has been 
started, results are not yet structured or specific 
enough to tell how to conduct training. With the Army's 
training population now spread so widely across the 
spectrum of aptitude, research is needed on the relation- 
ship of training performance to aptitude in order to 
determine what, if any, differential training is required 
for the efficient production of relatively standard MOS- 
qualified soldiers. 

The relationship between apticude level and training per- 
formance must be clarified before recommendations for 
increasing training efficiency can be made. This re- 
port presents research aimed at pT^>viding this informa- 
tion. Specifically, this report deals with the relation- 
ship between aptitude level and the acquisition of 
military skills and knowledges in a variety of training 
tasks which differ in complexity.— ^wt^z. 

McFann, Howard H. "Individualization of Army Training,'* 
in Innovations for Training, HumRRO Professional Paper 
6-69, The George Washington University, Human Resources 
Research Office, HumRRO Division No. 3, Presidio of 
Monter«.y, Calif., February 1969. (HamRRO presentation 
at the USCONARC Training Innovations Conference, Fort 
Benning, Ga., September 1968.) 

Individualization of training is discussed from the as- 
pects of (a) systems engineering, (b) training stra- 
tegies, and (c) individual training factors. Emphasis 
is on the latter two, recognizing that the systems en- 
gineering approach is a prerequisite for any training 
system. Four possible training strategies are discussed, 
including the implications of each for handling indivi- 
dual differences. Training factors to be simultaneously 
considered in developing a training program that 
handle individual differences are described. An attempt 
is made to interrelate ability level to factors of type 
of complexity of content, organization and sequencing, 
material, method and media of instruction, motivation, 
and management. — Auth. 
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INNOVATION More Coiranon Core Courses in Selected Subject Matter Areas 

PROBLEM Electronics is basic to many different Air Force speci- 

alties, and a number of slightly different courses have 
been designed. Because of their similarity, there is 
some duplication of effort in the design of these courses • 

SOLUTION Modify the Army's Common Core Courses for Basic Electron- 

ics courses (COBET) for Air Force use, 

ADVANTAGES Less duplication of effort in the design of electronics 
courses; improved quality of electronics training, 

RESOURCE The Army's course could be adapted for Air Force use 

REQUIREMENTS with approximately one man-year of effort- 

SELECTED Educational Computer Corporation, Final Report of 

REFERENCES SNAP/SMART^ prepared for Manpower Development Training, 
Department of Labor (Contract No, 82-40-67-56; 1 June 
1967-31 August 1969); and Department of Health, Educa- 
tion, and Welfare (Contract No, OEC-0-8-008096-2622 (089) ; 
16 January 1968-31 October 1969), 

Eschenbrenner, J, (Principal Investigator) and Lonigro, 
J. (Associate Investigator). "Advanced Development Work 
Resulting in Precision Measuring Equipment (PME) Indi- 
vidualized Instruction Materials," in Work Unit Sum- 
maries^ Defense Documentation Center, Defense Supply 
Agency, Alexandria, Va,, DDC Report No, CT9150, 22 
August 1972, p. 31, (Performing Organization: McDonnell 
Douglas Corp,, St. Louis, Mo,; Contract No, F3361S-71-C- 
1846,) 

Johnson, Frank F,, Jr, "Better Learning Management: 
COBET Offers a Model," American Vocational Journal^ Vol, 
47, No, 4, April 1972, 

Kratochvil, Daniel W. and Thompson, Lorna J, (American 
Institutes for Research, Palo Alto, Calif,), The Cluster 
Concept Program (Developed by the University of Maryland, 
Industrial Education Department) , Product Development Re- 
port No. 18, prepared for Office of Education, Office of 
Program Planning and Evaluation, Department of Health, 
Education, and Welfare, AIR-21900-l/72-TR(18) , January 
1972. 
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Adaptation of Air Force Training Materials for Distri- 
bution by State Boards of Vocational and Technical 
Education 

The Air Force invests a great deal of time and effort in 
keeping its technical training materials up to date-- 
much more so than any State could possibly afford. Yet, 
the civilian training programs have just as much need 
to remain up to date as the military. 

It would seem to be in the best interests of the country 
to at least make Air Force technical training materials 
available to State Boards of Vocational and Technical 
Education. Ideally, some federal agency (e.g., HEW) 
would go beyond this point and fund work on the civilian- 
izaticn of selected Air Force training materials for use 
by State Boards of Vocational and Technical Education. 

Less duplication of effort in the economy; improved 
training in civilian occupations that relate closely 
to Air Force needs and interests; increased transfer- 
ability of civilian skills. 

The cost of civilianizing CDC volumes is estimated from 
$10-$20 thousand each. A similar program has been tried 
by the Aerospace Education Fndn with promising results. 

The selection of courses for civilianization should be 
made by an organization concerned with national train- 
ing needs in the civilian sector. 

Aerospace Education Foundation. Vocational Instruction 
Systems of the Air Force Appliea to Civilian Education^ 
Praeger Publishers, New York, 1971, 128 pp. 

Mayo, G. Douglas. Programmed Instruction in Technical 
Training:, Research Report SRR 69-28, Naval Personnel 
Research Agency, San Diego, Calif., 1 June 1969. 
(AD 690 897) 

Shirley-Smith, K. '*The Armed Forces," Chapter 22 in 

Prograirw^A Learming in Integrated Industrial Training, 
Cower Press, London, 1968, pp. 173-178. 
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